LLM Fundamentals

NLP → Transformer → training pipeline → applications → model selection

Scroll reference — matches course-present*.html slide deck

Why did LLMs emerge?

From NLP to LLM: why traditional AI was not enough

Two questions to start (problem-driven)

NLP in one sentence: everything is prediction

Every NLP task = a prediction problem

Prediction loop (train and inference)

Four failures of traditional AI

Long-range dependency: Germany example

Language Model: mathematical definition

Chain rule: modeling language = modeling next-token probability, repeatedly.

GPT approximates $\mathcal{P}(s_{t+1} \mid s_{1:t})$ with a Transformer.

Numeric example (toy vocabulary {I, love, Paris})

If $\mathcal{P}(\text{I})=0.02$, $\mathcal{P}(\text{love}\mid\text{I})=0.12$, $\mathcal{P}(\text{Paris}\mid\text{I love})=0.09$:

Joint probability = $0.02 \times 0.12 \times 0.09 = \mathbf{0.000216}$

Tokenization & Embedding

Word embedding similarity

Semantically related words have similar vectors. Classic example:

Embeddings encode semantic relationships (gender, geography, tense) as geometric directions in vector space.

Why Transformer changed the world

Before formulas: understand why Attention beats RNN and CNN

RNN vs CNN vs Attention

Transformers also power computer vision and audio (ASR/TTS) — see Ch9.

Attention intuition: "Who matters most?"

Scaled dot-product attention

Self-attention: when $Q,K,V$ all come from input $X$:

In practice, learnable projections $W_Q, W_K, W_V$ are applied first:

• Q Query: what am I looking for?
• K Key: how matchable is each token?
• V Value: what information do I take?
• sqrt(d_k): scaling to prevent softmax saturation

Hand calculation (3 tokens: cat, sat, mat)

Scores for "sat": cat=0.85, sat=1.41, mat=0.28

Softmax weights: [0.26, 0.58, 0.16]

Output = $0.26 V_{cat} + 0.58 V_{sat} + 0.16 V_{mat}$

Multi-head attention

Each head uses its own $(W_Q^{(i)}, W_K^{(i)}, W_V^{(i)})$. Heads can focus on syntax, coreference, or semantics in parallel, then merge via $W_O$.

Transformer: Encoder-Decoder Architecture

Vaswani et al. 2017 — Attention Is All You Need

Three model families

Encoder-Decoder architecture diagram

Encoder layer internals

Decoder layer internals (3 sub-layers)

Positional encoding (sin/cos)

Attention has no built-in order; positional encoding injects token position. Modern models often use RoPE (rotary) instead, which extends better to long context.

GPT = Decoder-only (ChatGPT path)

No encoder stack. Your prompt is left context; model generates rightward with causal mask.

How data flows in Transformers

Pre-training vs Fine-tuning

How an LLM answers your question

Step-by-step decode for: "What is the capital of France?"

Seven-step inference loop

1. Tokenize → embed + positional encoding
2. Forward through L Transformer layers
3. LM Head → logits (vocab size |V|)
4. Softmax → $\mathcal{P}(\text{next token} \mid \text{context})$
5. Pick token (greedy / top-p) → "Paris"
6. Append → repeat until <EOS>

Numeric decode (toy logits)

Next: Temperature, top-p, decoding strategies

Inference parameters: temperature, top-p, top-k

Same model, different settings → very different output behavior

Core parameters

Temperature effect (Paris example logits)

Presets by task

Why "intelligence" suddenly emerges

Emergent abilities & Scaling Laws

Scaling Laws

Loss decreases predictably with parameters N, data tokens D, and compute C.

Continue: What is an LLM → training and apps first; model selection at the end.

What is an LLM?

Transformer models with 10B+ parameters trained on massive text

Five key techniques

Open-source LLM family

Modern LLM Training Pipeline

What happens at each stage, how it is done, and how we evaluate

Seven-stage pipeline overview

GPT family timeline

What each stage produces

Continue to Pretrain, SFT, RLHF, Eval for full detail.

Pretraining: how it actually works

Objective

Every position: predict next token from all previous tokens. Wrong prediction updates all weights via backprop.

End-to-end workflow

Data mix (typical)

Distributed training (how 175B is possible)

• Data parallel: same model on each GPU, different batches, average gradients
• Tensor parallel: split weight matrices across GPUs in one layer
• Pipeline parallel: split layers across GPUs
• Mixed precision (BF16): faster, less memory

Evaluation after pretraining

SFT / Instruction Tuning

Teach the model to follow instructions

Data format

<|user|> What is the capital of France?
<|assistant|> The capital of France is Paris.

Training procedure

1. Load pretrained weights (not random init)
2. Format as chat template
3. Mask loss on prompt — only train on assistant reply tokens
4. Train 1-3 epochs on 10k-1M examples, LR ~1e-5

Fine-tuning methods

Evaluation after SFT

RLHF and DPO

Ouyang et al. 2022

Why SFT alone fails

• SFT data is small; many valid answers exist
• Model can be verbose, toxic, or hallucinate confidently
• Need: helpful, honest, harmless (HHH)

RLHF three-step flowchart

PPO: Proximal Policy Optimization — limits policy change per step.

DPO: trains directly on preference pairs (y_w preferred over y_l) without separate reward model.

Evaluation after RLHF

Agent-era training and full evaluation

Stages 4-7

Master evaluation matrix

ChatGPT 3-step summary

Why LLM cannot be an Agent alone

The brain still needs arms, legs, and memory

Soul diagram: Agent = LLM + execution system

How agents are trained

Behavior training, not just prompts

1. Tool Calling — learn when/how to call APIs
   User: NYC weather? → {"tool":"weather_api","city":"New York"}
2. Planning — research → collect → analyze → report
3. Reflection — "Did I answer wrong? Should I retry?"
4. ReAct — Thought → Action → Observation loop

ReAct loop

Path: ③ Agent → ④ Retrieval → ⑤ LangChain/LangGraph → ⑥ MCP/A2A → ⑦ OpenClaw → ⑧ Multi-agent.

Long context & memory

Context Engineering

AI Coding Agent needs in one window:

• Entire repository
• Git history and PR comments
• API documentation
• Error logs and stack traces

Implementation: ⑤ SqliteSaver + Store · ⑧ Multi-agent (course path, not this appendix number).

Applications of LLMs

Real-world domains beyond chat

Model selection and deployment

After you understand training, agents, and applications — now pick the right model and settings

1. Match model to task (multimodal? Chinese? self-host?)
2. Set temperature / top-p for the task (see Ch2 sampling)
3. Know hard boundaries (cutoff, hallucination, context)
4. A/B test on real queries before production

Frontier model comparison · Capability boundaries

OpenAI vs DeepSeek vs others (2025-2026)

Frontier models compared — specs from official docs; benchmarks vary by task

Frontier model comparison

GPT-4o details

• Released May 2024: multimodal (text, audio, image in/out)
• Context window: 128K tokens; knowledge cutoff: Oct 2023 (per OpenAI model card)
• Max output: ~16K tokens (standard); Realtime variant uses 32K context

When to choose which?

Benchmark snapshot (indicative, not absolute)

Benchmarks shift with each model version. Use as rough guide; always eval on your task.

Model capability boundaries

What LLMs cannot do reliably — even frontier models

Seven hard boundaries

Closed vs open model tradeoffs

PM decision framework

1. Define task + eval set before picking a model
2. Check: need multimodal? real-time data? self-host?
3. Run A/B on 50-100 real user queries
4. Add RAG/tools where boundaries hit (cutoff, hallucination, action)
5. Monitor production: thumbs down, badcase replay, cost per task

The full picture

Traditional AI: input → output

LLM: input → reason → generate

Agent: goal → plan → act → feedback → iterate

NLP core: everything is next-token (next-unit) prediction.

RAG: offline index build + online retrieve-then-predict.

Multimodal: same prediction idea for vision (ViT, SAM, Diffusion) and audio (Whisper ASR, TTS).

Models: pick by task (GPT-4o multimodal, DeepSeek cost/reasoning, Llama self-host) — know the boundaries.

LLM is the brain. Agent is the whole person.