Search for a command to run...
No comments yet. Be the first to comment.
Every agentic AI system that "looks things up" before answering — a support bot searching a knowledge base, a coding agent searching a codebase, a research assistant searching PDFs — is running a Retr
RAG search has four moving parts: split the document, turn text into vectors, store those vectors somewhere queryable, and pull the right ones back out at query time. This post covers the last two — c
RAG systems live or die on one decision that happens before any embedding is computed: how the source document gets cut into pieces. This post walks through three chunking mechanisms available in Lang
This is Part 2 of an ongoing series on building a production-ready PostgreSQL AI agent using LangChain and Ollama. If you haven't read Part 1, start there — it covers the foundation: connecting to Pos
On this page
You implemented a search algorithm in your RAG system and it's returning data without any error, and you think that's it, you're done. You're wrong.
That mindset works fine for a normal system — an API, a script, a CRUD endpoint. No error, no crash, ship it. But RAG isn't that. Your search can run clean, return a full result set, zero exceptions, 200 all the way — and still be quietly handing your model the wrong chunks. "It didn't crash" tells you nothing about whether it actually found the right stuff. For that you need a different set of metrics, ones built specifically to answer "is my retrieval actually relevant," not "did my code run."
So in this post we're going to break down exactly those metrics, one by one, with real examples:
Precision@K — of what you retrieved, how much was actually useful
Recall@K — of everything relevant out there, how much did you actually find
MRR — how fast did you find the first relevant chunk
MAP@K — how good is the ranking across all relevant chunks, not just the first one
NDCG@K — how close to a perfect ranking, when some chunks are more relevant than others
And once retrieval's covered, we'll also get into how it connects to the generation side — faithfulness, answer relevance — plus how to build a golden dataset and use an LLM as a judge to actually score all this stuff automatically.
When a RAG system gives a wrong answer, there are two places it could have gone wrong, and conflating them is the single most common mistake in RAG debugging:
A bad answer could mean the retriever never found the right chunk (a retrieval failure) — in which case no generation model on earth could have answered correctly, it never saw the information.
Or it could mean retrieval did its job and handed over the right chunk, but the model ignored it, misread it, or made something up anyway (a generation failure). These need separate metrics because the fix is completely different: one is a search-tuning problem, the other is a prompting problem.
This article covers only retrieval side metrices in depth.
Every metric below is a different way of scoring the exact same thing: a ranked list of retrieved chunks, checked against a hand-labeled set of chunks that are actually relevant.
Four of the eight retrieved chunks are relevant, sitting at ranks 2, 4, 5, and 7. Every metric in this post is computed from this one picture — what changes is which ranks it looks at, and whether it cares about order at all.
Precision@K — of the K chunks you retrieved, how many are relevant?
Recall@K — of all relevant chunks that exist, how many did you actually find?
Neither metric cares where in the top-5 a relevant chunk landed — only whether it's inside the cutoff. That's their strength (simple, cheap to compute) and their weakness (a system that buries its one good chunk at rank 5 scores the same as one that puts it at rank 1, as long as both make the cutoff).
Recall@K only ever goes up as K grows. Push K to the size of your whole corpus and recall@K hits 1.0 trivially — you "found" everything because you retrieved everything. Always report recall at a realistic, fixed K (the K your reranker actually forwards to generation), and pair it with precision so a shameless K doesn't hide a bad system.
These two are less about ranking quality and more about two very different questions — one about noise, one about coverage — so it's worth reading them separately rather than as a single combined signal:
Precision@K is a noise question: of everything you're handing to the generation model, how much of it is actually useful? Low precision means your context window is full of filler — chunks the model has to read past (or worse, get distracted by) to find the real answer. This matters most when you're feeding a fixed, small K into generation, since every irrelevant chunk in that K is wasted context budget.
Recall@K is a coverage question: out of everything relevant that exists in your corpus, how much of it did retrieval even manage to surface? Low recall means the answer might not even be possible — no amount of good prompting or reranking can save you if the right chunk never made it into the candidate set at all. This is the metric that catches retrieval-stage blind spots, like an embedding model that consistently misses a certain phrasing of a question.
Reading them together: high precision + low recall means retrieval is being too conservative — it's confident about the few chunks it returns, but missing others that exist. Low precision + high recall means the opposite — it's casting a wide net and catching everything relevant, but drowning it in noise. Neither number alone tells you which failure mode you're in; you need both.
Because neither metric is order-aware, they're best used as a first-pass health check — "is my candidate set roughly the right shape" — before reaching for MRR, MAP@K, or NDCG@K to judge whether that candidate set is ranked well. A system can have great precision@K and recall@K and still produce a bad user experience if the one relevant chunk out of five is sitting at rank 5 instead of rank 1 — which is exactly the gap the next section covers.
MRR (Mean Reciprocal Rank) looks at one thing per query — the rank of the first relevant chunk — and scores it as 1 / rank, then averages across queries.
MRR is order-aware but first-hit-only — for the RRF query above, chunk_12 at rank 2 gives RR = 0.50, and MRR never looks past that point. Whatever happens at ranks 4, 5, and 7 (the other three relevant chunks in this example) is invisible to it. That blind spot is exactly what the next metric fixes.
A single MRR number is a proxy for "how much does the user have to scroll before they hit something useful." That makes it less an abstract IR score and more a direct read on system usability:
MRR close to 1.0 — your top result is usually the right one. Users (and your generation step) rarely need to look past rank 1.
MRR around 0.5 — the right chunk typically shows up around rank 2, on average. Retrieval is finding it, but it's not confident enough to put it first.
MRR trending low (below ~0.3) — either the right chunk is buried deep, or it's missing from the top-K entirely for a chunk of your queries (remember, a query with zero relevant hits in the top-K contributes an RR of 0, dragging the average down hard).
Because MRR only credits the first hit, it's the metric to watch specifically when your downstream generation step only reads the first chunk or two before answering, or when you're tuning a reranker and want a fast, cheap signal for "is the single best answer floating to the top." It's a poor fit, though, for judging whether your system surfaces all the relevant context for a question — for that, keep reading.
MAP@K (Mean Average Precision) computes precision@k at every rank where a relevant chunk shows up, averages those numbers into a per-query score called Average Precision, then means that across all queries.
💡 Key point: on Query 1, MRR scored 0.50 (it stopped at rank 2) while AP scored 0.54 (it credited all four hits). If a reranker change moved chunk_22 from rank 7 up to rank 3, MRR wouldn't move at all — chunk_12 is still the first hit at rank 2 — but AP (and MAP@K) would go up, correctly rewarding the improvement. Use MAP@K whenever queries can have more than one relevant chunk, which is the normal case once a corpus has multi-chunk sections
MAP@K is a single number that summarizes ranking quality across your entire golden set, in a way that's sensitive to both "did we find the relevant chunks" and "did we rank them near the top." That combination makes it the metric most people report as the headline number when comparing two retrieval systems end-to-end.
MAP@K close to 1.0 — nearly every relevant chunk for nearly every query is ranked at the very top. This is rare in practice and worth double-checking your golden set isn't too easy.
MAP@K in the 0.5–0.7 range — a healthy, typical score for a working hybrid search + reranker setup: most queries get their relevant chunks reasonably high, but there's room to improve on the harder queries.
MAP@K trending low — either relevant chunks are scattered deep in the ranking across many queries, or several queries are missing relevant chunks from the top-K entirely (each contributes an AP of 0, and those zeros drag the mean down fast).
Because MAP@K averages over every relevant chunk per query and every query in the set, it's the metric to reach for when you want one number to track over time as you tune chunking, embeddings, or reranking — a MAP@K that moves up after a change is a much stronger signal than a single query "looking better," since it means the improvement generalized across your whole golden set rather than fixing one query while quietly breaking another.
Okay so here's the thing every metric so far has been ignoring: relevance isn't actually binary. A chunk isn't just "relevant" or "not relevant" — some chunks nail the answer, some are kinda-sorta related, and some are just noise that happens to share a keyword.
Precision@K, recall@K, MRR, MAP@K — none of them can tell the difference between "this chunk perfectly answers the question" and "this chunk mentions the topic in passing." NDCG@K (Normalized Discounted Cumulative Gain) is the metric built specifically to care about that difference.
To use it, every chunk in your golden set gets a graded relevance score instead of a yes/no — say, 0 (irrelevant), 1 (tangential), 2 (partial answer), 3 (nails it). Here's what the actual math looks like on 3 chunks with relevance scores 3, 2, and 0, shown across three different orderings:
Let's walk through what's actually happening here, because the log2 stuff looks scarier than it is:
First, figure out the best possible ranking. Sort your chunks by relevance, highest first — that's the "Ideal" lane at the top. Compute its DCG (more on that formula in a second), and call that number IDCG. It's your ceiling — the best score any ordering could ever get.
Then score whatever ordering retrieval actually gave you. Same formula, just applied to the real ranking. Each chunk's relevance score gets divided by log2(rank + 1) — so a relevance-3 chunk sitting at rank 1 contributes a full 3.0, but that exact same chunk sitting at rank 3 only contributes 1.5. Same chunk, same "goodness," way less credit — because it's buried where fewer people will actually see it.
Divide actual by ideal. That's NDCG. In Ordering A, retrieval nailed the ideal ranking exactly, so DCG matches IDCG and NDCG = 1.00 — perfect score. In Ordering B, retrieval buried the single best chunk (relevance 3) all the way at rank 3, so DCG drops to 2.76 and NDCG drops to 0.65.
The whole point: NDCG doesn't just ask "did you find the good stuff," it asks "did you put the best stuff first." Two systems can retrieve the exact same three chunks and still get very different NDCG scores, purely based on the order.
⚠️ Gotcha: NDCG needs graded relevance labels (e.g. 0–3) on every golden example, not just relevant/not-relevant. That's real extra annotation work on top of a binary golden set. It's the industry-standard metric for web search ranking, but it's reasonable to treat as future work until the golden dataset has graded labels — don't block a first eval pass on it.
Basically, NDCG is the metric for when "we found the relevant chunk" isn't good enough on its own and you actually care about quality of match, not just presence.
NDCG@K close to 1.0 — your retrieval isn't just finding relevant stuff, it's putting the best stuff right at the top, close to the ideal ordering. This is what you want feeding into generation, since the model's most likely to lean on whatever's in the first slot or two.
NDCG@K noticeably lower than your MAP@K — this is the tell that your system is finding relevant chunks (MAP looks fine) but not distinguishing great chunks from okay chunks in how it ranks them. A reranker tuned only on binary relevance can hit this exact gap.
NDCG@K trending low across the board — either your best chunks are consistently getting buried, or your relevance grading itself needs a second pass (inconsistent grading tanks NDCG fast since it's so sensitive to the relative ordering of scores).
Use it when you've got graded relevance to spend on, and especially when "close enough" chunks genuinely aren't as good as the perfect one — which, if you're citing sources or answering precise technical questions (exactly your case), is basically always.
None of the above metrics mean anything without a trustworthy golden set to check against — and the construction choices here matter more than people expect.
Every single metric in this post — precision@K, recall@K, MRR, MAP@K, NDCG@K, even the faithfulness/relevance judge prompts — is computed relative to the golden set. There's no such thing as "precision" in the abstract; it's always "precision against these specific labeled examples." Which means the golden set isn't a one-off checklist item you build once and forget — it's the ruler you'll keep re-using every time you touch retrieval
Two rules worth holding firm on:
Hand-label, don't LLM-generate. LLM-generated Q&A pairs tend to reuse the source chunk's exact phrasing, which inflates retrieval scores relative to how a real user would actually ask. It also introduces self-grading bias if the same model later judges its own generated questions.
Reference real chunk_ids from the live table, not paraphrased summaries. This keeps the golden labels valid as ground truth even as embedding models or rerankers get swapped out underneath.
20–30 diverse examples — mixing factual lookup, comparison, "how does X work," and known edge cases (like a section that got split across chunk boundaries) — is enough for a credible first benchmark. More examples matter less than covering different query shapes.
| Metric | Order-aware? | Needs graded relevance? | Answers |
|---|---|---|---|
| Precision@K | No | No | Of what I retrieved, how much is useful |
| Recall@K | No | No | Of what's out there, how much did I find |
| MRR | Yes | No | How high did the first relevant hit rank |
| MAP@K | Yes | No | How good is the ranking across all relevant hits |
| NDCG@K | Yes | Yes | How close to ideal ranking, weighted by relevance strength |
The practical starting point: precision@K, recall@K, and MRR for a first pass, MAP@K once queries can have multiple relevant chunks, and NDCG@K once the golden set has graded relevance labels rather than binary ones.
