Why Retrieval Quality Decides RAG Quality

The bottleneck is upstream of the model

Retrieval-augmented generation (RAG) answers questions by retrieving relevant documents from a knowledge base and passing them to a large language model as context. When a RAG system gives a wrong answer, the cause is more often a retrieval failure — the right document was never retrieved, was chunked so its meaning was split, or was ranked below the cutoff — than a reasoning failure in the language model. Improving the model rarely fixes these systems; improving retrieval usually does.

That claim is the organising idea of this course. You built a working RAG pipeline in AI401 and learned the mechanics of embeddings and vector search in ML202. This course is about what happens next: the pipeline works in the demo, ships to production, and then answers confidently and wrongly — and you need to find out why.

An anatomy of a wrong answer

Consider a support assistant over a product knowledge base. A user asks:

"Does the Teams plan include SSO?"

The assistant answers: "Yes, the Teams plan includes single sign-on." The actual policy: SSO is Enterprise-only. Nothing in the generation step malfunctioned. Walk the pipeline backwards:

1. Generation — the model answered faithfully from the context it was given. The retrieved chunks described SSO setup and the Teams plan's features, in separate fragments. The model stitched them together.
2. Ranking — the chunk that says "SSO is available on the Enterprise plan only" existed in the index. It ranked 9th. The pipeline passed the top 5.
3. Chunking — the pricing page was split mid-table, so the chunk containing "Enterprise plan only" lost the header row naming the feature. Its embedding no longer resembled the query.
4. Ingestion — the pricing page had also been re-published a month earlier; both versions were in the index, and the older one said something different.

Four stages, four distinct failure modes, none of them visible in the answer. The answer just looks fluent. This is why RAG systems fail differently from classic ML systems: a classifier that degrades shows up in its metrics; a RAG system that degrades still produces confident prose.

Why "just improve the prompt" doesn't work

AI assistants, asked to fix a RAG system, overwhelmingly reach for the last stage: a better system prompt, a bigger model, more retrieved chunks. These are the levers visible in the code that assembles the final call. The levers that actually move answer quality — chunk boundaries, hybrid lexical-plus-semantic search, re-ranking, index freshness — live in code the assistant was often never shown, and in decisions that were never written down as code at all.

Passing more chunks is the most tempting non-fix. It sometimes helps, but it dilutes the model's attention, raises cost per call, and — critically — hides the retrieval problem instead of fixing it. If the right chunk is ranked 40th, no affordable top-k will save you, and you still won't know your ranking is broken.

What this course covers

The course follows the pipeline in the order failures compound:

• Module 1 — Retrieval Foundations, Deepened: chunking as a design decision, hybrid BM25-plus-dense retrieval with reciprocal rank fusion, and query rewriting (HyDE, multi-query, decomposition).
• Module 2 — Ranking & Evaluation: cross-encoder re-ranking at production scale, and the retrieval metrics — recall@k, MRR, nDCG — plus faithfulness and groundedness evals that let you measure instead of guess.
• Module 3 — Production RAG Systems: index freshness and embedding drift, permission-aware and multi-tenant retrieval, and observability and cost.
• Module 4 — Advanced Patterns: agentic, iterative retrieval — and a capstone where you audit a complete RAG pipeline written by an AI assistant.

Where ML202 taught you what embeddings encode and how vector search works, and AI401 taught you to treat retrieved context as part of a token budget, this course treats retrieval as its own engineering discipline with its own metrics, its own failure catalogue, and its own audit protocol.

The professional skill

Every lesson here answers the same implicit question: what do you need to understand to catch what AI gets wrong? AI assistants write fluent chunking code, plausible FAISS setups, and evaluation scripts that measure the wrong thing. The durable skill is diagnosing which stage of the pipeline an error came from and knowing which lever fixes it — because that diagnosis is exactly what the fluent output hides.

---

Summary: RAG quality is bounded by retrieval quality. Wrong answers usually trace back through ranking, chunking, and ingestion — not generation — and the symptom (fluent, confident prose) gives no signal about which stage failed. This course treats retrieval as an engineering discipline: design decisions, measurements, and an audit protocol for AI-generated pipelines.

Next: The first and most underestimated design decision — chunking, and why the "obvious" fixed-size split that AI tools default to quietly caps your recall.