Almost every RAG proof of concept succeeds. A worrying share of RAG products quietly underperform for months before anyone measures them. The gap is not the model, and it is not the vector database.
The demo is not representative
A proof of concept uses a curated handful of clean, current, non-contradictory documents that someone chose because they answer the questions being demonstrated. Production has none of those properties.
- Scanned PDFs where the text layer is missing or wrong.
- Three versions of the same policy, two of them superseded, none of them labelled.
- Documents that only some users are permitted to see.
- Questions whose answer is genuinely not in the corpus at all.
Chunking is the decision nobody revisits
Most teams pick a chunk size early, get acceptable demo results, and never look at it again. But chunking strategy has to suit your document shapes. A contract with deeply nested clauses does not chunk like a support ticket thread, and splitting a table across two chunks makes both meaningless.
This is unglamorous work with a large effect on answer quality, and it is where we spend a disproportionate share of RAG effort.
Pure vector search is not enough
Embeddings capture meaning and lose precision. Ask about a specific part number, error code, or policy reference and semantic search will confidently return something conceptually adjacent and factually wrong.
Hybrid retrieval — vector search combined with keyword matching, then reranked — handles both. If you are running vector-only search over a corpus containing identifiers, that alone may explain your quality problem.
Permissions have to be enforced at retrieval
This is the failure with real consequences. If your index contains documents the current user is not allowed to read, and permissions are applied after retrieval or in the interface, then a well-phrased question can surface content that person should never see.
Not knowing is a feature
A system that always produces an answer will confidently produce a wrong one when the corpus does not contain the information. Users learn this quickly, and once they do, they stop trusting the correct answers too.
Saying so plainly, and citing sources so a user can verify, is worth more to adoption than a marginal improvement in answer quality.
Without evaluation you are guessing
The single most common gap: no way to tell whether the system is getting better or worse. Someone adjusts a prompt, re-indexes, or the model version rolls forward, and quality shifts with nobody noticing until complaints arrive.
- 1Build a test set of real questions with known correct answers — fifty good ones beat a thousand generic ones.
- 2Score retrieval and answer quality separately, because they fail for different reasons.
- 3Run it on every change to prompts, chunking, models, or the index.
- 4Track how often the system correctly declines to answer.
What to do first
If you have a RAG system live and no evaluation, build the eval set before changing anything else. You cannot improve what you are not measuring, and most teams discover their quality is materially different from what they assumed.