Chapter 2

RAG

Look up first, then think.

Not a definition lecture — start with why AI needs external memory

Teaching order

Do NOT start with...

Embedding

Vector DB

Similarity Search

Students get lost if you open with infrastructure. Start with three real problems.
Problem 1

Why does ChatGPT "make things up"?

It predicts plausible text — not verified facts.

Hallucination: confident answers with no ground truth in the prompt.
Problem 2

Why doesn't GPT know your company docs?

Your internal PDFs, policies, and databases were never in pretraining.

Enterprise data cannot (and should not) all be baked into model weights.
Problem 3

Why does GPT knowledge go stale?

Training cutoff — e.g. knowledge frozen at Oct 2023.

Model weights are not a real-time database. Parameters ≠ live storage.
The answer

RAG = plug external knowledge into the LLM

Look up first, then think.

(Say this in class — not "retrieval-augmented generation")

Key diagram

RAG: Before vs After

Why AI must have an external memory system

BEFORE: LLM only AFTER: LLM + external memory User → LLM → Answer (?) Hallucination · no company data · stale knowledge Weights = frozen snapshot, not a live database User Knowledge Store docs, policies, wiki retrieve LLM Grounded Answer Look up first, then think AI must plug in an external memory system
Upgrade

RAG is not just "searching documents"

It is the foundation of an Agent Memory System.

Memory typeWhat it isImplementation
Short-termCurrent conversationContext window
Long-termRetrievable knowledgeVector DB (RAG index)
EpisodicPast tasks & sessionsSession / task logs
SemanticStable domain factsCurated knowledge base
Key diagram

Agent + RAG Architecture

LLM + Memory + RAG + Agent — finally connected

Agent + RAG Architecture User / Task LLM (Reasoning Brain) next-token prediction Planning decompose task Tools API, code, browser Action execute in world MEMORY SYSTEM (RAG lives here) Short-term Context window Long-term Vector DB (RAG index) Episodic Past tasks & sessions Semantic Stable domain knowledge RAG retrieve
One line each

Pretraining teaches knowledge.
RAG teaches retrieval.
Agents teach action.

Now: mechanics

Two phases (after students understand WHY)

PhaseWhenPlain English
OfflineBefore launchBuild the library (index docs once)
OnlineEvery questionLook up, then LLM thinks
OFFLINE: Docs → chunk → embed → store No generation — just building memory ONLINE: Query → retrieve → prompt → answer Look up first, then think
Phase 1

Build the knowledge index

  1. Load documents (PDF, wiki, DB export)
  2. Clean & chunk into passages
  3. Turn text into searchable vectors (embedding)
  4. Store in vector database
This is long-term / semantic memory — not magic, just organized storage.
Phase 2

Answer a question

  1. User asks a question
  2. Look up — find relevant passages in the index
  3. Think — LLM reads evidence + question, generates answer
  4. Return answer with citations
System: Answer ONLY from context below.
Context: [doc_42] EU refunds within 14 days...
User: What is our EU refund policy?
Technical (last)

Under the hood (optional depth)

$$\text{sim}(q,d)=\frac{\mathbf{v}_q\cdot\mathbf{v}_d}{\|\mathbf{v}_q\|\,\|\mathbf{v}_d\|}$$

Embedding maps text → vectors. Similar meaning → nearby vectors → retrieval works.

MetricOfflineOnline
QualityRecall@kFaithfulness, citation hit rate
Chapter 2 complete

Look up first, then think.

Scroll: course-full-rag.html

Next: ③ Agent Core · Deep retrieval: ④ Retrieval
9-chapter hub

Intro
1 / 1 Hub←→ Space F