Your RAG keeps citing the wrong document? Fix the chunking, not the model.

If your retrieval system keeps handing people a confident answer stapled to the wrong source, stop shopping for a bigger model. The model isn't the problem. Retrieval is, and nine times out of ten the real fix is how you cut your documents into chunks.

We build RAG for a living. Frank doesn't guess. He cites. So when a system cites the wrong doc, we don't reach for a new model. We open the pipeline.

Why does a smarter model still cite the wrong doc?

Because the model only sees what retrieval hands it. RAG is two jobs bolted together: find the right text, then write from it. The LLM does the writing. It has no say in the finding. If your retriever pulls the wrong chunk, a smarter model just writes a more convincing wrong answer. That's the trap. A model upgrade makes the mistake sound better, not go away.

Stack Overflow put it plainly: with RAG, you create embeddings of the pieces of data you want to draw from. Here's the part that bites. Those same pieces are what gets handed back as the citation. Get the pieces wrong and the citation points at the wrong thing. A citation is only worth something if it points at the right paragraph.

What breaks first?

Chunking. It's the least glamorous part of the stack, and it's where most systems bleed.

Here's the failure. You dump a 40-page PDF into a splitter set to some round token count. No overlap. No structure. The splitter cuts every N characters and doesn't care what it's cutting. It slices mid-sentence, mid-table, mid-idea. Now one “chunk” is the back half of one policy and the front half of the next. Embed that and you get a vector that means nothing in particular.

Stack Overflow said it clean. Include too much in a chunk and the vector loses the ability to be specific to anything it discusses. Include too little and you lose the context of the data. Too big, the chunk is a blur. Too small, it's a fragment with no home. Either way the retriever grabs something that looks related and isn't. The wrong doc wins.

So what's the fix? Cut smarter, not bigger.

Sensible chunk sizes with overlap, then structure-aware splitting. Start there.

Overlap is the cheap win. Instead of starting each chunk where the last one ended, you slide the window so neighboring chunks share a little text at the seams. Stack Overflow calls it a sliding window, and it keeps the context around the edges of each chunk from getting guillotined. It costs you some storage. It buys you chunks that still make sense at the borders.

Then stop splitting on raw character counts. Split on the document's own structure. Break on headings, paragraphs, list items, code blocks. Context-aware chunking uses the punctuation and markup already sitting in the text, periods, paragraph breaks, markdown, HTML, to decide where a meaningful piece starts and stops. A section stays a section. A code block stays whole instead of coming back broken. That last one matters more than people think. Stack Overflow's own example: hand raw code markdown to a naive recursive chunker and you get back broken code.

One more, from the people who build vector databases for a living. Smaller, semantically coherent units that map to a potential user query tend to retrieve better than fat ones. Chunk for the question, not for the page.

Where does metadata come in?

It's how you stop the retriever from searching your whole pile on every query. Tag every chunk. Source document, section, date, doc type, whatever your business actually filters on. Stack Overflow's advice for improving a RAG system is direct: filter your vector store results against metadata values so you only pull from the subset that matters. Want a how-to and not a contract? Filter on doc type. Now you've shrunk the search space before similarity even runs. Fewer wrong docs in the running means fewer wrong docs cited.

While you're in there, look at your embeddings. A weak embedding model buries the right chunk under noise. Try a few different embedding models and see which one actually performs best on your data. Don't assume the default is the right one for your domain. Test it.

Isn't retrieval solved once the chunks are clean?

Almost. Two more moves and you're most of the way home: hybrid search with reranking, then contextual retrieval.

Hybrid search runs keyword matching alongside vector similarity, so an exact term like a part number or a policy code doesn't get lost in fuzzy semantic space. Reranking then takes your top candidates and reorders them by real relevance before the model ever sees them. This is where Anthropic's numbers get loud.

Anthropic published a technique called Contextual Retrieval. The idea is simple. Before you embed a chunk, you prepend a short blurb that explains where that chunk sits in the full document. The chunk stops being an orphan. In their tests, contextual embeddings alone cut the top-20 retrieval failure rate by 35 percent. Add contextual BM25 keyword matching and it's a 49 percent drop. Layer a reranking step on top and failed retrievals fall by 67 percent. That's two thirds of your wrong-doc problem gone, and not one line of it is a bigger generation model. It's all retrieval.

Read that again. Sixty-seven percent fewer retrieval failures, from fixing how chunks get stored and ranked. The model stayed the same.

What does this mean if you're running a business?

It means the invoice matters. A bigger model is a bigger bill on every call, forever. Better chunking is a one-time engineering job on your pipeline. One is rent. The other is a repair. When your AI cites the wrong policy to a customer, the answer isn't “spend more per token.” It's “go look at how you cut the documents.”

It also means you can trust the thing again. An AI that cites the right source is an AI you can put in front of customers and auditors. One that confidently cites the wrong source is a liability wearing a nice font.

Frank doesn't play fetch with the wrong stick and call it a win. He brings back the right one.

The bottom line

Wrong citations aren't a model problem. They're a plumbing problem. Fix the chunks. Add overlap. Split on structure. Tag with metadata. Run hybrid search and a reranker. Give each chunk its context before you embed it. Do that and the wrong-doc problem mostly disappears, on the model you already pay for.

Bigger model, same broken retrieval, louder wrong answers. Skip it. Fix the pipe.

Got a RAG system citing ghosts? Start the brief.