Evaluation Guide: all 9 chapters

What to measure, when, and where to find depth

RAG evaluation

• Faithfulness — answer supported by retrieved chunks (RAGAS)
• Citation hit rate — cited chunk IDs match gold doc IDs
• Golden set — question + expected doc IDs + answer snippet

Component eval: measure retrieval Recall@k before end-to-end faithfulness.

Agent evaluation

• Tool accuracy — correct tool + valid JSON args
• Task success — final state matches expected outcome
• Steps to success — penalize unnecessary loops
• Golden set — 20 queries with expected tool sequence

Retrieval evaluation

• Recall@k / MRR — gold doc in top-k?
• Context recall — gold answer inferable from retrieved text
• Ablation table — dense vs BM25 vs hybrid vs +rerank

LangGraph / memory evaluation

• Resume rate — same thread_id continues without re-asking facts
• Store recall — new thread_id still loads user preferences
• Interrupt compliance — dangerous tools pause until approved

MCP & A2A evaluation

• MCP tool latency — p95 call time per tool
• Schema adherence — args match inputSchema
• A2A handoff success — Task completes with valid Artifact

OpenClaw & Hermes evaluation

• Tool JSON validity rate — Hermes vs base model on 5 prompts
• Skill trigger accuracy — right skill for user intent
• End-to-end latency — message to Telegram reply

Multi-agent & KG evaluation

• Handoff accuracy — supervisor routed to correct worker
• Critic pass rate — reviewer accepted output without rewrite
• KG hop recall — multi-hop question answered via Cypher path
• Token cost — multi-agent vs single agent on same task

Multimodal evaluation

• WER (ASR) · IoU (segmentation) · FID (generation)
• MOS (TTS quality) · VQA accuracy (vision-language)

Recommended eval workflow

1. Define task — one sentence user goal per test case.
2. Build golden set — 20–50 cases with expected outcome (doc ID, tool, answer snippet).
3. Component eval first — retrieval before end-to-end RAG; tools before full agent.
4. Automate — RAGAS, jiwer, custom scripts in CI on every index/model change.
5. Human spot-check — 10% sample weekly; focus on failures.
6. Regression gate — block deploy if Recall@5 or task success drops >2%.

Decision tree: where is the bug?