Retrievalยท7 min read
RAG for Agents Without the Buzzwords
When to give an agent retrieval, how to do it simply, and the failure modes that make RAG look broken.
RAG for Agents Without the Buzzwords
Retrieval-Augmented Generation just means: before the agent answers, fetch relevant text and put it in context. That's it.
When you actually need it
- The knowledge changes faster than you can retrain or re-prompt.
- The corpus is too big to paste into context.
- You need citations back to a source.
If none of those are true, a well-written system prompt may beat a vector database.
The simple version that works
- Split documents into chunks of a few hundred tokens.
- Embed the chunks and store them.
- On each query, embed the question, pull the top few chunks, and hand them to the model with the instruction to answer *only* from them.
Failure modes that look like "RAG is broken"
- Chunks too big โ you retrieve a whole page to answer one line, and the model drowns.
- No re-ranking โ the closest vector isn't always the most relevant; a quick re-rank pass fixes most "why did it miss the obvious doc" complaints.
- No "I don't know" โ without permission to say the answer isn't in the sources, the model invents one.
Start simple, look at what it retrieves, and only add machinery when a real query fails.