Multi-agent & Knowledge Graph

Collaboration patterns + structured knowledge

Multi-agent collaboration patterns

When one agent + many tools is not enough

Frameworks: LangGraph (recommended), AutoGen, CrewAI, OpenAI Agents SDK.

Supervisor pattern in LangGraph

from langgraph.graph import StateGraph, START, END

class TeamState(TypedDict):
    messages: Annotated[list, add_messages]
    next: str  # which worker to call

def supervisor(state: TeamState):
    # LLM decides: researcher | writer | reviewer | FINISH
    decision = llm.invoke(SUPERVISOR_PROMPT + str(state["messages"]))
    return {"next": parse_next(decision)}

def researcher(state): ...
def writer(state): ...
def reviewer(state): ...

builder = StateGraph(TeamState)
builder.add_node("supervisor", supervisor)
builder.add_node("researcher", researcher)
builder.add_node("writer", writer)
builder.add_node("reviewer", reviewer)
builder.add_edge(START, "supervisor")
builder.add_conditional_edges("supervisor", lambda s: s["next"],
    {"researcher": "researcher", "writer": "writer", "reviewer": "reviewer", "FINISH": END})
for worker in ["researcher", "writer", "reviewer"]:
    builder.add_edge(worker, "supervisor")  # report back

Critic-reviewer loop (simpler variant)

Max 3 critic rounds; else escalate to human.

Hands-on lab: supervisor team

1. Task: "Write a 1-page brief on Q2 sales trends" with CSV data file.
2. Researcher agent: tool = run_python (pandas describe); returns stats JSON.
3. Writer agent: no tools; consumes stats; outputs markdown brief.
4. Reviewer agent: checks numbers match stats; returns pass/fail JSON.
5. Supervisor: routes until reviewer passes or 3 iterations.
6. Compare: single agent with all tools vs supervisor team on quality + token cost.

Knowledge Graph fundamentals

Structured facts as nodes & edges — not text chunks

Triple model (Subject — Predicate — Object)

(Alice) --[REPORTS_TO]--> (Bob)
(Project-X) --[OWNED_BY]--> (Alice)
(Project-X) --[USES]--> (PostgreSQL)

Vector RAG vs Graph retrieval

Building a knowledge graph from documents

Offline pipeline

1. Ingest — load PDFs, wikis, tickets into raw text.
2. Extract — LLM or NER model outputs triples per chunk.
3. Normalize — merge "Bob Smith" / "Bob" / "b.smith@co.com" into one entity ID.
4. Load — upsert nodes/edges into Neo4j, NebulaGraph, or NetworkX (dev).
5. Index (optional) — embed entity descriptions for hybrid entity search.

LLM extraction prompt (minimal)

SYSTEM: Extract (subject, predicate, object) triples from the text.
Return JSON array. Use UPPER_SNAKE predicates.

TEXT: "Alice Chen leads Project Phoenix, which runs on PostgreSQL.
       She reports to Bob, VP of Engineering."

OUTPUT:
[
  {"s":"Alice Chen","p":"LEADS","o":"Project Phoenix"},
  {"s":"Project Phoenix","p":"RUNS_ON","o":"PostgreSQL"},
  {"s":"Alice Chen","p":"REPORTS_TO","o":"Bob"}
]

Load into Neo4j (Cypher)

MERGE (a:Person {name: "Alice Chen"})
MERGE (p:Project {name: "Project Phoenix"})
MERGE (db:Tech {name: "PostgreSQL"})
MERGE (b:Person {name: "Bob"})
MERGE (a)-[:LEADS]->(p)
MERGE (p)-[:RUNS_ON]->(db)
MERGE (a)-[:REPORTS_TO]->(b)

Tools: LLMGraphTransformer (LangChain), GraphRAG indexer (Microsoft), Neo4j LLM KG Builder.

Graph retrieval at query time

Agent chooses: vector search, graph traverse, or both

Pattern A: Text-to-Cypher

LLM converts natural language → Cypher query → execute on Neo4j → results as context.

# User: "What database does Alice's project use?"

Cypher:
MATCH (a:Person {name:"Alice Chen"})-[:LEADS]->(p:Project)-[:RUNS_ON]->(db)
RETURN p.name, db.name

# Result → inject into LLM prompt → natural language answer

Pattern B: Entity linking + subgraph

1. Detect entities in query ("Alice", "Project Phoenix")
2. Link to graph node IDs (fuzzy match or embedding)
3. Pull k-hop neighborhood as structured context
4. LLM answers from subgraph JSON

Pattern C: Hybrid (recommended)

async def retrieve(query: str):
    vector_hits = await vector_db.similarity_search(query, k=5)
    entities = extract_entities(query)
    graph_hits = neo4j.run(subgraph_query(entities, hops=2))
    return merge_and_deduplicate(vector_hits, graph_hits)

GraphRAG (Microsoft pattern)

Community summaries + global/local search for corpus-scale graphs

Two-phase architecture

When GraphRAG beats naive vector RAG

• "What are the main themes across all compliance reports?" → global community summaries
• "How is Team A connected to the 2024 audit finding?" → local subgraph + hops
• 1000+ docs with repeating entity names → entity resolution + graph wins

Hands-on lab: graph-enhanced RAG

Dataset (use any internal wiki export or sample)

10–20 short pages about people, projects, and systems. Enough for 3-hop questions.

1. Baseline vector RAG — chunk + embed + answer 5 test questions. Log failures on relationship questions.
2. Extract triples — run LLM extraction on each chunk; inspect JSON quality manually.
3. Load Neo4j (Docker: neo4j:5) — MERGE triples; visualize in Browser.
4. Add Cypher tool to your LangGraph agent from LangChain & LangGraph lab.
5. Re-run same 5 questions — compare faithfulness on multi-hop queries.
6. Hybrid — combine vector + graph retrieval; measure Recall@5 on a labeled set.

Eval questions (example)

Multi-agent evaluation

Team-level metrics

KG / GraphRAG eval

# Multi-hop KG eval
def multi_hop_recall(query_entities, gold_entities, subgraph_fn, hops=2):
    nodes = subgraph_fn(query_entities, hops)
    found = sum(1 for g in gold_entities if g in nodes)
    return found / len(gold_entities)

Retrieval metrics: Retrieval Engineering. Full matrix: Evaluation Guide.