A RAG Answer Can Be Grounded and Still Be Wrong
Retrieval-augmented generation, or RAG, pairs a generative model with a separate knowledge base. The system retrieves information for a query and places it in the model’s context. That is the current NIST definition of RAG, and it preserves the central idea of the original NeurIPS RAG paper: combine parametric model knowledge with retrievable, non-parametric evidence.
Retrieval makes source-based answers possible. It does not make them automatically true.
A response can be faithful to a retrieved document that is expired. It can cite the correct policy and misread its exception. It can retrieve excellent evidence that the user was never authorised to see. It can answer only half a compound question. Groundedness therefore cannot stand in for correctness, completeness, security or usefulness.
A retrieval quality audit separates the pipeline into testable layers so the team fixes the smallest broken component instead of repeatedly changing the prompt or model.
Use an Eight-Layer Failure Map
| Layer | Audit question | Typical failure |
|---|---|---|
| Source truth | Does an authoritative, current answer exist? | No owner, conflicting policies, missing attachment |
| Ingestion | Was the source extracted and indexed correctly? | Broken table, OCR error, lost heading, empty page |
| Identity and access | Is this user allowed to retrieve it? | Cross-tenant leak, stale group membership, post-retrieval filtering |
| Query understanding | Did the system represent the real information need? | Acronym mismatch, lost conversation context, poor query rewrite |
| Candidate retrieval | Did relevant evidence enter the candidate set? | Low recall, wrong metadata filter, dense-search miss |
| Ranking and context | Was decisive evidence ranked and assembled well? | Old version above current one, duplicate chunks, evidence buried in context |
| Generation | Did the answer use the evidence correctly and completely? | Unsupported inference, omitted exception, failed abstention |
| Citation and action | Can the user verify and safely act on the result? | Dead link, wrong location, inaccessible citation, unsafe downstream action |
This decomposition reflects current evaluation practice. NIST’s 2025 TREC RAG track evaluates passage retrieval, augmented generation and the full RAG system separately. Microsoft’s current RAG guidance likewise recommends documenting granular retrieval and embedding results as well as end-to-end system results: large-language-model RAG evaluation.
Use the resulting claims, test versions, and limitations as entries in the AI assurance evidence-pack framework, rather than leaving them in an isolated engineering dashboard.
Build a Golden Set From Real Work
A useful evaluation set is not a list of easy FAQs generated from the same documents being tested. Sample real questions, failures, searches and escalations, then have domain owners label them.
Each test case should contain:
| Field | What to record |
|---|---|
| Query | The exact user question, including relevant conversation history |
| Intent | What task or decision the user is trying to complete |
| Expected answer | A reference answer or required answer points |
| Retrieval ground truth | Documents or chunks judged relevant—often called qrels |
| Minimum evidence set | Evidence without which the answer is incomplete |
| Excluded evidence | Expired, wrong-jurisdiction or otherwise invalid sources |
| Identity context | User, role, tenant, groups and relevant entitlements |
| Valid time | The date or policy version for which the answer is correct |
| Abstention expectation | Whether the system should answer, clarify, refuse or escalate |
| Risk and slice labels | File type, language, department, customer group and impact |
Include positive and negative cases. Negative cases should cover questions outside the corpus, users without permission, malicious documents, ambiguous requests and questions for which sources conflict. Microsoft’s information-retrieval evaluation guidance explicitly recommends both positive and negative examples; a system that always returns its least-bad chunks can look active while failing to recognise “no answer”.
Stratify the set by the material differences in your estate: clean HTML, scanned PDFs, spreadsheets, tables, tickets, short policies, long contracts, languages, document age and permission model. Report every important slice as well as the aggregate.
Audit the Corpus Before Comparing Retrievers
No search configuration can recover a source that is absent or unreadable. Measure:
- source coverage against the authoritative inventory;
- parse success by file type;
- empty or near-empty extraction rate;
- table, footnote and heading preservation;
- duplicate and near-duplicate rate;
- records without an owner, effective date or review date;
- source-to-index refresh time;
- deletion and permission-revocation propagation time;
- version conflicts and expired-document rate.
Open samples from the search index and compare them with the source—not merely the ingestion status. Standard OCR can flatten headings, tables and lists and destroy relationships needed for retrieval. Google’s current Document AI layout parser guidance documents this failure mode and uses structure-aware chunks with ancestral headings. Treat vendor claims as features to evaluate on your files, not proof that your own extraction is correct.
Store stable source identifiers, version, effective date, owner, jurisdiction, sensitivity and access metadata with each chunk. A citation should be traceable back through the chunk to the exact source version.
Test Permissions as a Security Invariant
Permission accuracy is not an ordinary relevance score. An average of 99.9% may conceal a serious data leak.
Construct allow-and-deny cases for each material role and tenant. Verify that restricted documents do not enter the candidate set, model context, citation list, logs or cache. Test group changes and access revocation, then measure how long the index takes to reflect them. Set a target of zero observed unauthorised retrievals in the defined suite and report the test count and limits; zero observed failures is not proof of zero residual risk.
The OWASP GenAI Security Project identifies vector and embedding weaknesses including unauthorised access, cross-context leakage, data poisoning and embedding inversion. Its mitigations include fine-grained permission-aware stores, logical separation, source validation and retrieval logging. Microsoft’s document-level access-control guidance describes query-time security trimming and also warns that permission changes can lag until metadata is synchronised.
Apply authorisation before or during retrieval. Removing forbidden chunks after retrieval is too late if they have already entered traces, rerankers, prompts or caches.
Measure Retrieval Independently
Judge the candidate set before looking at the generated prose.
| Metric | What it answers | Important limitation |
|---|---|---|
| Recall@k | What proportion of known relevant evidence appeared in the top k? | Requires sufficiently complete relevance labels |
| Precision@k | What proportion of the top k was relevant? | Treats all relevant items equally |
| MRR | How early did the first relevant result appear? | Ignores additional evidence after the first hit |
| nDCG@k | Did the system rank highly relevant evidence ahead of weaker evidence? | Requires graded relevance judgements |
| No-answer false-positive rate | How often did retrieval return plausible material when no valid answer existed? | Needs realistic negative queries |
| Current-version hit rate | Was the valid version preferred over expired or superseded sources? | Requires version metadata and temporal labels |
Microsoft Foundry’s current RAG evaluator documentation distinguishes retrieval evaluation using ground-truth relevance labels from final-answer groundedness, relevance and completeness. That is the right separation.
“Rank” by itself is not a complete metric. Report a defined measure at a defined k, on a versioned dataset, with sample counts and slices.
Compare Search Strategies as Controlled Experiments
Use the same corpus snapshot, queries and judgements to compare:
| Run | Candidate method | Reranker | Filters | Purpose |
|---|---|---|---|---|
| A | Keyword/BM25 | None | Identical ACL and validity filters | Robust lexical baseline |
| B | Dense vector | None | Same | Test semantic matching |
| C | Hybrid keyword + vector | None | Same | Test combined recall |
| D | Hybrid | Semantic or cross-encoder reranker | Same | Test ranking quality and latency trade-off |
The BEIR benchmark found that BM25 remains a robust baseline, while reranking and late-interaction approaches performed strongly on average at higher computational cost. Do not interpret that result as a universal winner; it is a reason to keep a lexical baseline and test on the target domain.
Azure AI Search’s hybrid-search documentation explains how keyword and vector queries run in parallel and their rankings are fused. OpenAI’s current vector-store search API similarly exposes metadata filters, query rewriting, reranking, score thresholds and result-count controls. Every one of those settings can change recall, latency and cost, so version them with the evaluation result.
Treat Chunking as a File-Type Decision
There is no universally correct chunk size. Small chunks can lose qualifying context; large chunks can dilute relevance and increase model input. Fixed overlap may help prose and repeat headers unnecessarily. Tables, slide decks, tickets and contracts need different treatment.
For each important file type, compare at least two plausible strategies while holding the retriever constant. Inspect whether chunks preserve section titles, table headers, dates, footnotes and source locations. Measure retrieval metrics and tokens placed into context. If changing chunking improves recall but doubles context with irrelevant text, the end-to-end result may worsen.
Document the parser, chunking rule, maximum size, overlap, metadata enrichment and embedding model. Re-run the set when any of them changes.
Evaluate the Answer on Separate Dimensions
Once retrieval is inside tolerance, assess the final response:
- Groundedness or faithfulness: are its claims supported by the supplied context?
- Correctness: are its claims factually and logically right?
- Completeness: does it cover every material part of the question or reference answer?
- Relevance: does it directly address the user’s need?
- Citation correctness: does each citation support the nearby claim?
- Citation completeness: are all material externally verifiable claims cited?
- Abstention quality: does the system stop or clarify when evidence is missing, conflicting or restricted?
A high-groundedness answer can still be wrong if the source is wrong or the model draws an invalid conclusion. Microsoft gives this exact diagnostic distinction in its end-to-end RAG evaluation guide.
Automated judges accelerate iteration, but calibrate them against human labels. RAGAS offers reference-free measures across retrieval and generation. ARES evaluates context relevance, answer faithfulness and answer relevance using lightweight judges plus a small human-annotated set. Neither removes the need for domain review of high-impact cases.
Also test context order. The peer-reviewed Lost in the Middle study found that several long-context models performed best when relevant information was near the beginning or end and degraded when it appeared in the middle. A large context window is not evidence that every supplied passage is used reliably.
A Worked Audit Trace
Question: “Can a UK sales employee claim a £240 conference hotel when the normal cap is £180?”
Ground truth requires the current UK travel policy and the conference-organiser exception rule. The 2024 policy is expired; the US policy is irrelevant; the user may view internal travel rules but not executive travel records.
Trace the result:
- Source: both current documents exist and have effective dates.
- Ingestion: the exception table was parsed with its column headings.
- Access: only permitted policy chunks entered retrieval.
- Query: “hotel cap” and “organiser-selected accommodation” were represented.
- Candidates: both required sources appeared in the top ten; the expired policy did not.
- Ranking: the exception appeared before general background.
- Generation: the answer stated the cap, exception, evidence required and approval route without inventing automatic eligibility.
- Citation: each rule linked to the exact current section and opened for the test user.
If the answer omits the exception because its chunk ranked 14th, tune retrieval or ranking. If the exception was in context but ignored, investigate context assembly and generation. If the current policy was never indexed, fix ownership or ingestion. The trace prevents a model upgrade from masking the real defect.
Monitor Changes, Not Just Traffic
Re-run the golden set after changes to parsers, chunking, embeddings, query rewriting, ranking, metadata, access rules, models, prompts or major document collections. In production, watch:
- no-result and low-score queries;
- invalid-source and expired-version retrieval;
- permission-denied and leakage-test results;
- citation-open failures;
- user corrections, reopened cases and escalations;
- retrieval and answer metrics by slice;
- p50 and p95 retrieval and end-to-end latency;
- tokens, searches and cost per successful answer;
- new sources dominating results;
- deletion and revocation lag.
Store enough of each evaluation trace to reproduce the decision: corpus snapshot, identity context, query and rewrite, filters, candidate IDs and scores, reranking, final context order, prompt/model version, answer and citations. Apply data-minimisation, security and retention controls to those traces.
To keep that observability proportionate, connect searches, tokens, reranking, review, and recovery to the unit economics in our production AI cost guide.
A trustworthy enterprise RAG system is not the one that always produces an answer. It is the one that retrieves authorised and current evidence, ranks the decisive material, uses it correctly, cites it precisely, and recognises when the record is not good enough to proceed.
