Retrieval Engineering

Standalone deep dive — teach after RAG concept, before production stack

RAG course explains why and offline/online flow. This course explains how to pick, tune, and measure retrieval.

BM25 (lexical / sparse retrieval)

Best for: SKUs, legal citations, exact product names, error codes, IDs.

Intuition

Score = term frequency in doc (saturated) × inverse doc frequency (rare terms weigh more) × length normalization.

$$\text{BM25}(q,d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t,d) \cdot (k_1+1)}{f(t,d) + k_1 \cdot (1 - b + b \cdot |d|/\text{avgdl})}$$

Code (rank_bm25)

from rank_bm25 import BM25Okapi

corpus = [doc.page_content.split() for doc in documents]
bm25 = BM25Okapi(corpus)

def sparse_search(query: str, k: int = 10):
    scores = bm25.get_scores(query.split())
    top = sorted(range(len(scores)), key=lambda i: -scores[i])[:k]
    return [documents[i] for i in top]

Semantic (dense) retrieval

Embed query and docs into vectors; rank by cosine similarity.

Model selection

Index + search (FAISS / Chroma)

from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

vs = FAISS.from_documents(chunks, OpenAIEmbeddings(model="text-embedding-3-small"))

def dense_search(query: str, k: int = 10):
    return vs.similarity_search_with_score(query, k=k)
    # returns [(Document, distance), ...]

Hybrid retrieval (BM25 + dense)

Run both retrievers in parallel; merge rankings.

RRF (Reciprocal Rank Fusion)

$$\text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)}$$

Typical $k=60$. No score normalization needed — robust when BM25 and cosine scales differ.

def rrf_merge(lists: list[list], k: int = 60, top_n: int = 10):
    scores = {}
    for ranked in lists:
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]

# Usage
sparse_ids = [d.id for d in sparse_search(q, 50)]
dense_ids  = [d.id for d in dense_search(q, 50)]
final_ids  = rrf_merge([sparse_ids, dense_ids], top_n=20)

Reranking: precision at the top

Bi-encoder retrieval is fast but approximate. Cross-encoder scores (query, doc) jointly — slower, sharper.

Two-stage pipeline

1. Hybrid retrieve Top-50 (cheap)
2. Cross-encoder rerank → Top-5 to LLM

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def rerank(query: str, docs: list, top_k: int = 5):
    pairs = [(query, d.page_content) for d in docs]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(docs, scores), key=lambda x: -x[1])
    return [d for d, _ in ranked[:top_k]]

End-to-end retrieval pipeline

def retrieve_for_rag(query: str) -> list[str]:
    # 1. Hybrid recall
    sparse = sparse_search(query, k=50)
    dense  = [d for d, _ in dense_search(query, k=50)]
    merged = rrf_merge([[d.id for d in sparse], [d.id for d in dense]], top_n=30)
    candidates = [doc_store[i] for i in merged]

    # 2. Rerank
    top = rerank(query, candidates, top_k=5)

    # 3. Optional: dedupe overlapping chunks, add metadata
    return [format_chunk(d) for d in top]

Selection guide

Retrieval & RAG evaluation

Build a golden set (minimum 30 Q&A pairs)

# golden.json
[
  {"question": "Refund window for electronics?", "gold_doc_ids": ["policy-sec-3"]},
  {"question": "SKU AX-9912 warranty?", "gold_doc_ids": ["sku-ax9912"]}
]

Retrieval metrics (code)

def recall_at_k(retrieved_ids, gold_ids, k=5):
    hit = any(g in retrieved_ids[:k] for g in gold_ids)
    return int(hit)

def mrr(retrieved_ids, gold_ids):
    for rank, doc_id in enumerate(retrieved_ids, 1):
        if doc_id in gold_ids:
            return 1 / rank
    return 0.0

End-to-end with RAGAS

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

result = evaluate(
    dataset=eval_dataset,  # question, answer, contexts, ground_truth
    metrics=[faithfulness, answer_relevancy, context_recall],
)
print(result)

Hands-on lab: retrieval ablation

Compare 4 configs on the same 20-question golden set

pip install rank_bm25 faiss-cpu sentence-transformers ragas langchain-openai

1. Prepare corpus: 50–100 markdown docs; chunk 512 tokens, 64 overlap; record chunk IDs.
2. Label 20 questions with gold_doc_ids (human annotation).
3. Config A: dense only → measure Recall@5, MRR.
4. Config B: BM25 only → same metrics. Note which question types each wins.
5. Config C: hybrid RRF → should beat A or B on mixed queries.
6. Config D: hybrid + rerank → measure Recall@5 and end-to-end faithfulness (RAGAS).
7. Report: table of configs vs Recall@5 / latency / faithfulness; pick production config.

Expected insight

Hybrid fixes sparse-only synonym misses; rerank fixes "right doc ranked #12" failures. If faithfulness still low after good Recall@5, the bug is in generation — not retrieval.