GraphRAG & Knowledge Graphs

Vector RAG retrieves chunks that are similar to the query. That works beautifully until the answer requires connecting facts that live in different chunks — "which projects did people who left the security team start afterward?" No single chunk contains that; the answer is a path through relationships. GraphRAG retrieves over a knowledge graph of entities and relationships instead of (or alongside) a flat vector index, so multi-hop and global questions become tractable.

Where Vector RAG Falls Down

• Multi-hop questions — answers requiring you to chain A→B→C. Vector search finds chunks about A and chunks about C but has no notion of the link between them.
• Global / aggregate questions — "what are the main themes across these 500 documents?" There's no single similar chunk; the answer is a property of the whole corpus.
• Relationship queries — anything about how entities connect rather than what a document says.
• Disambiguation — "Apple" the company vs the fruit; flat embeddings blur entities that a graph keeps distinct.

How GraphRAG Works

1. Extraction — run an LLM over the corpus to pull out entities (people, orgs, concepts) and the relationships between them, producing a graph. This is the expensive, offline step.
2. Community detection — cluster the graph into communities and pre-summarize each one. These summaries answer global questions without touching every document.
3. Retrieval — at query time, two modes:
   • Local search — find the query's entities, then traverse their neighborhood (related entities, connecting facts) to gather context.
   • Global search — route the question across community summaries and aggregate, for corpus-wide questions.
4. Generation — feed the retrieved subgraph or summaries to the model as context, same as ordinary RAG.

Cost and Tradeoffs

GraphRAG is not a free upgrade — it's a different cost profile:

• Expensive indexing. Extracting a graph means an LLM pass over the whole corpus, often more. Indexing cost can be orders of magnitude higher than embedding the same text.
• Extraction quality is the ceiling. Garbage entities and missed relationships flow straight through to retrieval. The graph is only as good as the extraction prompts and the source data.
• Staleness and maintenance. Updating the graph as documents change is harder than upserting a vector. Incremental graph updates are an active pain point.
• Overkill for most apps. If your questions are "what does the docs say about X," flat vector RAG is cheaper and just as good.

The Pragmatic Middle Ground

Most production systems don't go full GraphRAG. Common hybrids:

• Vector + metadata filtering — attach structured fields (author, date, type) to chunks and filter before vector search. Covers many "relationship-ish" queries without a real graph.
• Lightweight entity linking — extract entities at query time, fetch their canonical records, inject as extra context alongside vector hits.
• Graph as a reranker / expander — retrieve with vectors, then walk the graph from the retrieved entities to pull in connected context the vector search missed.

What to Watch For

Don't reach for a knowledge graph because it sounds sophisticated. Diagnose first: log the questions your vector RAG fails, and check whether they're genuinely multi-hop/global or just bad chunking and weak reranking. Most "RAG isn't working" problems are solved by better chunking, hybrid search, and a reranker — all far cheaper than standing up and maintaining a graph. Reach for GraphRAG when you've confirmed the failures are relational, not retrieval-quality.