RAG That Actually Retrieves
A case study on rebuilding a retrieval pipeline that was quietly returning the wrong context — and the 3x recall improvement that followed.
The assistant sounded confident and was often wrong. The model was fine — it was being fed the wrong context. This is a story about fixing retrieval, not generation.
The Problem
Users asked precise questions and got plausible but incorrect answers. Every investigation ended at the same place: the right document was in the database but never made it into the prompt.
Investigation
I built a tiny labeled set of question-to-document pairs and measured recall@5. It was 31%. Two-thirds of the time, the correct source was not even in the candidate set.
- Chunks were too large, blurring distinct concepts.
- Pure vector search missed exact-match terms like error codes.
- No re-ranking meant near-misses stayed near-misses.
The Solution
Smaller semantic chunks, a hybrid of vector and full-text search in Postgres, and a lightweight re-ranker on the merged candidates.
select id, content,
(0.6 * (1 - (embedding <=> $1))) +
(0.4 * ts_rank(fts, plainto_tsquery($2))) as score
from documents
order by score desc
limit 20;
The Result
Recall@5 went from 31% to 94%. Answer accuracy followed. We changed nothing about the generation model — the fix was entirely upstream.
Lessons Learned
- Build an eval set before optimizing anything.
- Hybrid search beats pure vectors for real questions.
- Chunking is a product decision, not a config value.
