Vector RAG and Graph RAG Retrieved Almost Nothing in Common

I wanted to know whether graph RAG is actually better than vector RAG, or whether “better” is the wrong question. The claim you see most often — that graphs beat embeddings on real corpora — is usually made without the two sides holding anything else constant, so it’s impossible to tell whether the graph won or the prompt did.

So I built both over one corpus and pinned everything downstream of retrieval.

The headline: on twelve questions, the two pipelines overlapped on a mean of 1.8 of 12 retrieved documents. They are nearly disjoint. And which one wins is predictable from the shape of the question, before either retriever runs — which means the useful output of a comparison like this isn’t a winner, it’s a routing rule.

The setup, and why it’s shaped this way

The corpus is a year of my own work mailbox — about 3,900 messages. That’s a deliberately awkward corpus: threaded, heavily duplicated, half of it automated notifications, and with real structure (who talks to whom, what replies to what) sitting right next to real prose. Both properties matter, because vector retrieval only sees the prose and graph retrieval only sees the structure.

The whole experiment rests on one constraint: both pipelines end in the same prompt template against the same model at temperature 0. Retrieval is the only free variable. If I changed the answer prompt, I had to change it for both sides in the same commit.

A single local 27B model does both the answering and, on the graph side, the query planning. Everything runs on one workstation plus one database host, which keeps the cost of a 12-question run near zero and made it cheap to re-run after every change, which mattered more than I expected.

The part that mattered most was not the retrieval

Before either index existed, the corpus had to be normalised, and that turned out to be where most of the real work was.

Two things about email that will break a naive ingest:

Quoted reply history. Outlook re-embeds the entire thread in every reply. If you index that, near-duplicate chunks dominate top-k: the same paragraph, quoted eleven times, filling your context window with one message pretending to be eleven. Chunking only each message’s new text took the corpus from 166 MB to 4.5 MB — a median of 646 characters of actual new content per message. That single decision did more for retrieval quality than any parameter I tuned afterwards.

Fields that lie. The export’s nominal plaintext body field frequently contains a complete HTML document, so markup has to be stripped unconditionally rather than only when falling back to the HTML field. And 189 messages had no sender, subject, or body at all — all of them carried the full RFC822 message in a different field, so those were recovered by parsing raw headers. Sender recovery went from 95% to 100%, which matters disproportionately on the graph side: a message with no sender is a message with no edges.

If you take one thing from this post and it isn’t the routing rule, take this: the ceiling on both pipelines was set during parsing.

The vector side

Conventional, on purpose. I wanted the graph compared against a competent baseline, not a strawman.

Two choices worth explaining:

Each chunk carries a header with its date, sender, and subject. An isolated chunk otherwise arrives at the model with no provenance, and the model then cannot tell you who said the thing it just found. Cheap fix, large effect on answer quality.

Chunks are retrieved but messages are returned. The top 40 chunks get collapsed to their best-scoring parent messages, and the top 12 messages go to the prompt. Retrieving chunks and answering from whole messages avoids the classic failure where the answer sits two sentences past the chunk boundary.

Retrieval takes about 60 ms end to end, embedding included. It is very hard to beat that on cost.

The graph side

Every edge here is derivable from message metadata. No LLM was used to build the graph.

That was the decision I went back and forth on most, so here’s the reasoning. A structural graph builds in about 60 seconds and every edge in it is verifiable — if the graph says two people exchanged several hundred messages, that number is a count of rows, not a model’s recollection. An LLM-extracted graph is far more capable and introduces a second thing that can be wrong. For a first comparison I wanted the graph’s contribution to be unambiguous.

Note also what the graph does not store: message bodies. Traversal returns a file identifier and the retriever joins back to Postgres for text. The graph contributes structure, not a second copy of the corpus. This kept it honest — any content the graph side used, it had to reach by structure first.

Retrieval is plan → link → traverse → expand:

The step with no analogue in top-k similarity is expand: once you have a few seed messages, pulling their thread siblings gets you the rest of the conversation whether or not those messages resemble the question. A reply that says only “agreed, let’s do that” is worthless to an embedding and essential to an answer.

The other thing with no vector analogue is graph facts — counts, correspondence weights, thread sizes and spans, computed in Cypher and handed to the model as ground truth rather than inferred from message text.

Two implementation notes for anyone using Apache AGE, both of which cost me real time: LOAD 'age'; must run once per session before the first cypher() call or the planner hook isn’t installed and you get an unhelpful unhandled cypher(cstring) function call. And AGE 1.5 has no relationship-type alternation — -[:TO|CC]-> is a syntax error, so you need -[r]-> with WHERE type(r) IN ['TO','CC']. Also count() cannot appear inside a map projection.

Cypher errors in the graph retriever are collected and surfaced in the UI rather than caught and ignored. A silently swallowed query error is indistinguishable from “no results found”, and that distinction hid two real bugs during the build.

What the comparison actually showed

Twelve questions, chosen before I knew what either side would do, spanning aggregate, structural, and content lookups. Results split into three clean classes.

Aggregate and structural questions: the graph wins outright. Ask “who did I exchange the most email with this year” and vector RAG correctly refuses — its answer is, in effect, “the retrieved context is a limited sample of twelve messages and cannot support a superlative.” That’s the right answer from twelve documents. The graph answers with an exact count, because the count is an edge property, not a sentence in any email. Same for “which threads had the most back-and-forth”: the graph returns message counts and date spans per thread; the vector side can only see that one retrieved message has five Re: prefixes.

Note the failure mode: vector RAG did not hallucinate a ranking. It declined. That’s the behaviour you want and it’s still a failure to answer a question the data could answer.

Content lookups that name no entity: the vector side wins outright. The worst case for the graph was a question of the form “what rate was quoted for the extra testing days?” — no person, no organisation, no date. The plan extracts nothing linkable, the graph falls back to matching keywords against subject lines, seeds four irrelevant messages and answers “not in context.” The vector side retrieves the paragraph and answers correctly. Subject-keyword fallback is the weakest path in the whole system and it’s the direct consequence of building the graph without content.

Entity-anchored content questions: a tie, by different routes. Ask about a payment schedule agreed with a named counterparty and both sides answer correctly, both citing the same message. The graph reaches it via organisation → sender domain → messages; the vector store reaches it by similarity. Same document, no shared mechanism.

On cost:

	vector	graph
retrieval	~60 ms	~200 ms Cypher + ~800 ms planning LLM call
end to end	4.0 s	5.5 s
context sent to model	~15k chars	~12k chars

The graph’s overhead is the planning call, not the traversal. If you want graph retrieval to be fast, that’s the thing to attack — a small local model or a cached classifier, not a better index.

One result I did not expect: across all twelve questions, neither pipeline cited a document it had not retrieved. Both retrievers are wrong plenty of the time; neither invented a source. With provenance-tagged chunks and a prompt that asks for citations, that class of hallucination just didn’t show up.

What I got wrong about the question

I went in asking which pipeline is better. That question has no answer, and not in the diplomatic “it depends” sense — the two pipelines fail on disjoint inputs, and you can tell which one will fail before you retrieve anything. The graph already produces a structured plan for every question. That plan contains the routing signal: does the question name a person, an organisation, a date range? Is the intent aggregate?

So the deliverable isn’t a winner. It’s this:

Two cheap pieces of that are already sitting unused in my build. A lexical index exists but nothing queries it — fusing BM25 with cosine by reciprocal rank is the standard fix for the exact-token failures that pure embeddings miss (case numbers, ticket IDs, surnames), and it’s an afternoon of work. A cross-encoder reranker over the top 40 candidates fits on the same spare GPU that already serves the embedding model. Neither requires touching the graph.

Third column, and the honest recommendation out of a comparison like this: graph-selected candidates reranked by embedding similarity. Structure picks the neighbourhood, semantics picks the document.

The bigger unlock, and its price

The structural graph knows who talked to whom, in which thread, when. It does not know what any message is about. That’s the entire explanation for the subject-keyword failure, and it’s fixable with one extraction pass over message bodies:

Measured cost on my hardware: about 2.1 s per message single-stream, so roughly 2.3 hours for the full corpus, and about 1.7 hours if you skip the automated mail — which contributes nothing but noise nodes anyway. Write it as a resumable batch job keyed by document, so a crash costs one message rather than the whole pass.

What it buys is a question class neither pipeline can currently touch: which engagements slipped, and who owns the next step? No single chunk contains that answer, so no top-k search can retrieve it. It requires joining a commitment to a person to a thread to a date. That is the strongest case for graph retrieval, and my build doesn’t make it — because I chose the version of the graph I could fully verify over the version that would have won the argument.

Below that on the list, in payoff order: hierarchical community summaries over the correspondence graph, so global “what were the themes this year” questions get answered from summaries rather than from a twelve-document sample — currently the weakest case for both pipelines. Then attachments, which are not ingested at all right now, meaning the pipeline only ever sees emails referring to documents rather than the documents. Then incremental ingest: both builders currently drop and recreate, which is fine at 3,900 documents and absurd at 40,000.

What this experiment does not show

It does not show that either pipeline is accurate. My evaluation harness measures timing, context size, retrieval overlap and citation counts. It has no ground truth, so it cannot score correctness — every judgement above about who “won” a question is me reading two answers and deciding, which is exactly the kind of evaluation I’d push back on if someone showed it to me.

Fixing that is a couple of hours of unglamorous labelling: 30–50 questions annotated with the documents that actually contain the answer, then recall@k and MRR per pipeline. That turns “the graph answer reads better” into a number. Until then the overlap statistic — 1.8 of 12 — is the one figure here I’d defend, because it’s mechanical and requires no judgement at all.

Which, conveniently, is also the finding that mattered.