AI
Why Your AI Gives Confident Wrong Answers About Your Own Documents.
By Muhammad UmarNovember 9, 20257 min readIssue #10
The model is not making things up. It answered correctly from the three paragraphs it was handed, and nothing checked whether those were the right three.
You point an assistant at your own documentation and ask it something you already know the answer to, as a test.
It answers confidently. It is also wrong, and the correct answer is definitely in there, because you wrote it.
The usual word for this is hallucination, and here it is almost always the wrong diagnosis. In most of these cases the model behaved perfectly. It was handed a few paragraphs and asked a question, and it answered from what it was given. The mistake happened before the model saw anything.
The problem: retrieval decides the answer, not the model
A model cannot read your whole document store on every question. Even where the context window would allow it, cost and latency usually will not.
So the standard arrangement is to split your documents into chunks, store each chunk with a numeric representation of its meaning, and at question time fetch the handful that look closest to the question.1 Those get pasted above your question, and that combined text is all the model ever sees.
Read that again with the failure in mind. The answer is chosen by the search step. The model is the last stage, and it can only work with what arrives. If retrieval hands over the wrong paragraphs, a better model produces a more articulate wrong answer.
Where the meaning gets cut
Chunking is usually done by size. Split every 500 tokens, or every 1,000 characters, with some overlap between neighbours.
Documents are not written in fixed-size pieces. So the split lands wherever it lands.
The policy says refunds are available within 30 days, and then says subscriptions are excluded. Cut between those two sentences and you have produced a chunk that is a confident, complete, wrong statement of your refund policy.
Nothing downstream can catch this. The chunk is not corrupted or malformed. It reads as a clean, authoritative passage, and it scores well against a question about refunds precisely because it is about refunds.
Every qualification lives after the claim it qualifies. Fixed-size chunking cuts exactly where meaning reverses.
Similar is not the same as relevant
The second failure is subtler and harder to spot in testing.
Embeddings place text in a space where things about similar subjects sit close together.2 That works well until you ask a question whose correct answer does not resemble the question.
Ask "which customers are on the old pricing?" and vector search will happily return your page titled Pricing, because that page is topically identical. The answer might live in a migration note that never uses the word pricing at all.
Two more cases where closeness misleads. Negation barely moves an embedding, so supports SSO and does not support SSO sit near each other. And a question phrased in the exact vocabulary of an outdated document will pull that document over its corrected replacement, because the newer version uses different words.
Nothing in the chain can report a problem
Retrieval returns the closest matches available. It has no notion of good enough, so it returns the top few whether they are excellent or hopeless.
The model then behaves as instructed. It was told to answer from the provided context, so it does, and it has no way to know that the paragraph containing the exception was never included.
That is the whole reason these failures are so convincing. Every component did its job, and the mistake sits in the gap between them where nobody is checking.
What actually helps
Chunk on structure, not on length
Split on headings, sections, or list items. Your documents already contain the author’s own decisions about where one idea ends, and a fixed character count discards all of that information for no benefit.
Prefix each chunk with its document title and heading path. A fragment reading "these are excluded" is useless alone and usable when it arrives labelled with the section it came from.
Wrong when your documents have no structure to speak of. Transcripts, chat logs, and scanned material give you nothing to split on, and there you are back to size with overlap.
Search with words as well as with meaning
Run a keyword search alongside the vector search and combine the results. Classical term-based ranking is very good at exactly what embeddings are worst at: names, error codes, product identifiers, and rare words that must match literally.3
This one change fixes a large share of real failures, because a great many questions contain a specific term that simply has to appear in the answer.
Wrong when your users ask conceptual questions in their own words with no shared vocabulary. Term matching contributes little there and you are adding a component for nothing.
Fetch more, then rerank
Retrieve twenty candidates rather than three, then use a model that reads the question and each candidate together to reorder them, and pass on the best few.
This works because the first stage compares two numeric summaries produced independently, while the reranker actually reads both texts side by side. It is the difference between matching topics and judging whether this passage answers this question.
Wrong when latency is tight or volume is high. Reranking adds a model call per query, and on a small well-structured corpus it may buy you nothing over just retrieving more.
Filter before you search
Store dates, versions, authorship, and status alongside each chunk, and exclude the ones that should not be eligible before ranking anything.
The superseded-document failure is not a search quality problem and cannot be fixed by better ranking. The old page really is the best match for the question. It should not have been a candidate.
Consider skipping retrieval entirely
If your entire corpus fits in a context window, put all of it in. Every failure above is a consequence of choosing what to include, and choosing nothing removes the category.
People skip past this option because a retrieval pipeline feels like the professional answer. For a handbook, a set of policies, or one project’s documentation, it is often both simpler and more accurate to hand over the lot.
| Change | Fixes | Cost | Skip it when |
|---|---|---|---|
| Structural chunking | Meaning cut mid-argument | A parser per document format | Your documents have no structure |
| Keyword plus vector | Names, codes, exact terms | A second index to maintain | Questions never contain specific terms |
| Reranking | Topically close, actually unhelpful | One model call per query | Latency matters more than the last few points of accuracy |
| Metadata filters | Confidently citing superseded pages | Metadata discipline at write time | Nothing is ever superseded |
| No retrieval at all | Every retrieval failure | Tokens per request | The corpus cannot fit |
How to find out whether yours is broken
The reason these systems ship broken is that the usual test is asking a few questions and reading the answers, which is the one method guaranteed not to detect the problem. A wrong answer built from a plausible passage reads exactly like a right one.
Two things work better, and neither is difficult.
Look at the retrieved chunks, not the answer. Log what was fetched for every question and read that instead. You will usually find the failure immediately, and you will find it in cases where the final answer happened to come out right anyway.
Write questions whose answers you know are qualified. Pick the rules in your documents that have exceptions, and ask about the exception. Those are the questions that break on chunk boundaries, and they are the ones nobody thinks to test because they feel like edge cases. They are not edge cases. They are most of a policy document.
When this is the wrong advice
When the documents are the problem. No retrieval system rescues a corpus that contradicts itself. If three pages describe the process differently and none is marked current, a perfect pipeline will faithfully return one of them. That is a documentation problem wearing a search costume.
When accuracy has to be guaranteed. Every technique above reduces the failure rate and none removes it. For medical, legal, or financial answers where being wrong is not recoverable, the output needs a human between it and the reader, and the system should be showing sources rather than conclusions.
When you have not measured anything yet. Adding reranking, hybrid search, and filters at once to a system nobody has evaluated means you will not know which change helped, and you will carry all three costs forever. Find the failures first.
The takeaway
When the answer is wrong, the instinct is to blame the model or reach for a larger one. Almost always the model never had the information, and a larger one would have been wrong more fluently.
Go and look at what was retrieved. That is where the answer was decided, and it is the one part of the system nobody watches.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020. The paper that named the pattern of retrieving passages and conditioning generation on them.
- Embeddings map text to vectors positioned so that related meanings sit near each other. The property that makes them useful, tolerance of different wording, is the same property that makes them weak on exact terms and on negation.
- BM25, from Robertson and Walker’s probabilistic retrieval work in the 1990s, remains the standard term-based ranking function and is what most search engines use as a baseline.