Book notes: the RAG chapter of AI Engineering

I built retrieval into two products before I could have given you a clean definition of it. Compounder runs hybrid search over an embedded canon of Buffett letters and filings. Mila answers dental front-desk questions out of a knowledge base sitting in pgvector. Both work. Then I read the RAG chapter of Chip Huyen's AI Engineering and had the slightly deflating experience of reading the manual for a machine I had already shipped. Some of what I built was right for reasons I could not have explained. Some of it was right by accident. These notes are about the difference.

Retrieval is older than the R in RAG

The thing the chapter fixed first was my sense of where the new part is. RAG feels new because the G is new. The R is not. Retrieval is ranking documents by how useful they are to a query, and search people have been doing that for decades. BM25, which is still the term-based baseline everything gets measured against, is older than Google. When I wired keyword search into Compounder alongside the embeddings, I thought I was adding a fallback. I was actually adding the senior partner.

That reframing matters because it tells you where to look when your system is bad. Everyone's instinct, mine included, is to blame the model or fiddle with the prompt. But if the right passage never made it into the context, the model never had a chance. The generator cannot be right about text it never saw. Most of the quality in a RAG system lives in the boring half, and the boring half has forty years of prior art.

Embeddings are not the whole answer

Embedding-based retrieval finds things that mean the same thing. That is its gift and its blind spot. Ask Compounder something conceptual, like whether a company's returns are durable, and semantic search earns its keep, because nobody in a 10-K writes the sentence "our returns are not durable." The signal is spread across paragraphs that never use your words.

But ask for something exact, a ticker, a specific ratio, a name, and embeddings go soft. Everything financial is vaguely similar to everything else financial in embedding space. A ticker symbol is not a concept. It is a string, and strings are what term-based search is for. The two methods fail in different directions, which is exactly why you run both and merge the results. I had built that without being able to defend it. Now I can: hybrid search is not a hedge, it is coverage of two different failure modes. On the risk desk we never accepted a single control for two different risks, and I should have recognized the shape sooner.

Chunking is where quality goes to die

The chapter is politely blunt about something I learned the annoying way: the least glamorous decision in the pipeline, how you split your documents, quietly sets the ceiling on everything downstream. Chunk too small and you retrieve sentences with no context around them. Chunk too big and the passage you hand the model is mostly noise with the answer buried somewhere inside, and you pay for every token of the noise.

What actually worked for me was chunking by structure instead of by count. Dental FAQs got split by question, because a question and its answer is the natural unit a caller's query maps to. Filings got split by section, because "Risk Factors" and "Management Discussion" are different documents wearing the same cover. Token-count chunking treats a document like a log to be sawed into equal pieces. Real documents have joints, and cutting at the joints is almost free quality.

Long context did not kill RAG

Every few months someone declares that million-token context windows make retrieval obsolete. Just paste everything in. The chapter walks through why that keeps not happening, and it matches what I see in my own systems. Stuffing the context costs money on every single call, adds latency, and buries the relevant passage in a haystack you built yourself. Models also do not attend to a huge context evenly. The answer being technically present is not the same as the answer being found.

The framing I keep now is that context is the expensive, fast tier and the document store is the cheap, slow tier, and retrieval is the policy that decides what earns promotion. That is a memory hierarchy. Nobody argued that bigger RAM made disks obsolete. You still have to decide what deserves to be in the fast tier, and that deciding is the entire job.

Evaluate the retriever like it's the suspect

The habit I took from the chapter that changed my systems most: score retrieval on its own, before the model ever touches the result. Did the right chunks come back, and how high did they rank. If recall is bad at the retrieval step, no amount of prompt work downstream will save you, and if you only ever eyeball the final answer, you will spend days interrogating the innocent component.

This one landed hard because the failure it prevents is the quiet kind, and quiet failures are my whole professional fear. A RAG system with a weak retriever does not crash. It hands back a confident, fluent answer assembled from the wrong material, formatted beautifully, and moves on. The worst answer a system can give is the wrong one delivered with good grammar. You do not catch that by admiring outputs. You catch it by auditing what the retriever fetched, the same way a desk audits fills and not just the P&L.

What the fundamentals actually bought me

None of this made my systems dramatically fancier. It made them explainable to me, which turns out to be the thing that compounds. When Compounder misses now, I know which half to interrogate first. When Mila's knowledge base grows, I know the chunking decision is a product decision and not a config value. And when the next architecture fad arrives claiming retrieval is dead, I have a way to think about it that does not depend on the fad's own marketing.

Fundamentals are unglamorous exactly until something breaks. Then they are the only thing you have. That was true on the desk, and the more of this book I get through, the more it looks true here too.

AI Engineering is on my reading shelf.