RAG: Look up first, then think

Course path (9 chapters)
  1. LLM
  2. RAG
  3. Agent Core
  4. Retrieval Engineering
  5. LangChain & LangGraph
  6. MCP & A2A
  7. OpenClaw & Hermes
  8. Multi-agent & KG
  9. Multimodal
Chapter 2

RAG

Look up first, then think.

Teaching order (important): Do NOT open with Embedding, Vector DB, or Similarity Search. Students will be lost.

Start with three problems every user feels. Then show why AI must plug in external memory. Technical details come last.

Matches course-present-rag.html slide deck.

Part 1 · Why

Why do we need RAG?

Three questions — problem-driven, not definition-driven

Question 1: Why does ChatGPT "make things up"?

LLMs predict plausible next tokens. They are not fact-checkers.

Hallucination: fluent, confident answers with no grounding in verified sources.

Question 2: Why doesn't GPT know your company documents?

Internal PDFs, wikis, CRM exports, and policies were never in pretraining data.

Enterprise / private knowledge cannot (and should not) all be baked into model weights.

Question 3: Why does GPT knowledge expire?

Every model has a training cutoff. Weights freeze at train time.

Model parameters are not a real-time database. You cannot "update" facts by hoping the LLM remembers.

Therefore

RAG = give the LLM an external knowledge system.

In class, say: "Look up first, then think." — not the academic translation "retrieval-augmented generation."

Key diagram

RAG: Before vs After

You are not defining RAG — you are explaining why AI must have external memory

BEFORE: LLM only AFTER: LLM + external memory User → LLM Answer (?) Hallucination · no company data · stale knowledge User Knowledge Store docs, policies, wiki LLM Grounded Answer Look up first, then think
Part 2 · Cognitive upgrade

RAG is not just "searching documents"

It is the core of an Agent Memory System

Memory typeRoleTypical implementation
Short-term memoryCurrent conversation, recent turnsContext window (128K tokens)
Long-term memoryRetrievable knowledge at query timeVector database (RAG index)
Episodic memoryWhat happened in past sessions / tasksSession logs, task history store
Semantic memoryStable facts about domain / userCurated knowledge base, user profile
This framing turns RAG from a "search trick" into a Cognitive AI topic — memory, not just retrieval.
Key diagram

Agent + RAG Architecture

LLM + Memory + RAG + Agent — connected in one picture

Agent + RAG Architecture User / Task LLM (Reasoning Brain) Planning Tools Action MEMORY SYSTEM Short-term Context window Long-term Vector DB (RAG) Episodic Task history Semantic Domain knowledge RAG retrieve
One line each

The big picture

Pretraining teaches knowledge.

RAG teaches retrieval.

Agents teach action.

RAG essence: Look up first, then think.

Part 3 · How (after WHY)

How RAG works

Introduce mechanics only after students understand the problem

Two phases

PhaseWhenPlain English
OfflineBefore launch; on doc updateBuild the library (index documents once)
OnlineEvery user questionLook up relevant passages, then LLM thinks
OFFLINE (build once) ONLINE (per query) Docs → chunk → embed → vector store No LLM generation — building memory Query → retrieve → prompt + evidence → LLM Look up first, then think
Phase 1

Offline: build the knowledge index

1. Load documents (PDF, HTML, wiki, DB export)
↓
2. Parse & clean (OCR if scanned)
↓
3. Chunk into passages (~256–512 tokens, overlap)
↓
4. Embed text into vectors (searchable representation)
↓
5. Store in vector database
↓
6. Validate: Recall@k on sample queries

FAISS / ChromaDB / Milvus / Pinecone / pgvector

Phase 2

Online: look up, then think

1. User asks a question
↓
2. Look up — find Top-k relevant chunks in index
↓
3. Build prompt: system + retrieved evidence + question
↓
4. Think — LLM generates answer (next-token prediction)
↓
5. Return answer + citations

Prompt template

System: Answer using ONLY the context below. Cite [doc_id].
Context:
[chunk_42] EU refunds allowed within 14 days...
User: What is our EU refund policy?

Technical detail (teach last)

$$\text{sim}(q, d) = \frac{\mathbf{v}_q \cdot \mathbf{v}_d}{\|\mathbf{v}_q\| \, \|\mathbf{v}_d\|}$$

Embedding maps text to vectors. Similar meaning → nearby vectors → retrieval finds the right passage.

Evaluation

PhaseMetrics
OfflineRecall@k, chunk quality, index build latency
OnlineFaithfulness, citation hit rate, end-to-end latency
Common mistake: Teaching online flow only. Without offline index, RAG fails silently.

Advanced retrieval (BM25, hybrid, rerank, eval): Retrieval Engineering course.

Advanced (summary)

Retrieval beyond naive vector search

Full depth in Retrieval Engineering course

MethodWhen to use
BM25Keywords, SKUs, exact terms
Dense semanticParaphrase, conceptual match
Hybrid + RRFProduction default
Cross-encoder rerankTop-50 → Top-5 precision

Metrics: Recall@k, MRR, NDCG, faithfulness, citation accuracy.