Ask an AI chatbot about your company's refund policy and it faces a choice: guess from whatever it picked up in training, or admit it has no idea. Hand it the policy handbook first and the answer changes: grounded in the handbook, with the relevant section there to check against.
The upgrade has a name, and it's one of the most common designs in applied AI: retrieval-augmented generation, or RAG. The model doesn't get smarter. It gets the right pages open before it answers.
What RAG is
RAG is a two-step design for answering questions: retrieve relevant text from a knowledge source, then generate the answer with that text in front of the model.
The name comes from a 2020 research paper by Patrick Lewis and colleagues, then at Facebook AI Research, which paired a language model with a retriever that searched a dense vector index of Wikipedia. The system set state-of-the-art results on three open-domain question-answering benchmarks, and its answers came out more specific and more factual than the same model answering alone. The recipe stuck. Versions of this design now sit inside many of the AI products you touch.
The distinction shows up in exams. A plain language model answers like a student in a closed-book exam, working from whatever soaked in during training. RAG turns it into an open-book exam: the book sits on the desk, and the model flips to the right page before writing anything.
Why RAG exists
A model's training knowledge is frozen when training ends. It can't contain your company handbook, yesterday's meeting notes, or a document published last week, and a model asked about them has to improvise.
Uploading a document into the chat window helps, and for a single file it's often all you need. Three limits show up as the pile grows:
- The window fills. Everything you send shares one context window, so a growing library competes with the conversation for the same space.
- Quality fades with length. A 2025 technical report that tested 18 models found accuracy tending to drop as input grows, even inside the window, and relevant details get harder for the model to use when they're buried in the middle.
- Answers lose their receipts. When the model draws on a paste of everything, it can't point to which document a claim came from.
RAG helps with all three: only selected passages enter the window, the system can keep track of where each passage came from, and the library behind the lookup can be much larger than a single context window.
How RAG works, step by step
RAG has two phases: a setup pass over the documents, then the same lookup with every question.
Setup, done once per collection:
- Chunk the documents. Long files get split into passages of a few paragraphs, big enough to make sense alone and small enough to mix well with others.
- Embed each chunk. An embedding is a list of numbers, a few hundred to a few thousand entries long, that represents what a passage means. Passages about similar topics get similar lists of numbers, so meaning becomes geometry: close meanings land at close coordinates.
- Store the embeddings in an index built to answer one question fast: which stored vectors sit closest to this one?
Then, with every question:
- Embed the question the same way the chunks were embedded.
- Retrieve. The system finds the chunks whose vectors sit closest to the question's vector and keeps the top few.
- Generate. Those chunks get placed in the model's prompt next to the question, and the model writes its answer from them.
The lookup works on meaning rather than keywords. Ask "how do I get my money back" and the system can retrieve a chunk titled "Refund policy," because the two texts mean the same thing even though they share few words. Keyword search would need the word "refund" to appear in your question. The vector lookup doesn't.
Where you already use RAG
You may already be using RAG when you upload a PDF to ChatGPT and ask about it: a retrieval step finds the relevant pages before the model writes. Add documents to a Claude Project and every chat in that project draws on them; Anthropic sized each Project around a 200K-token context window, the equivalent of a 500-page book. NotebookLM builds its whole experience around sources you upload, and search tools that answer with citations run retrieval over the web before generating a word.
We run a small version of the same pattern in this blog's pipeline. Every morning a script scores the day's reader comments into themes, then checks the winning theme against the posts already published here before anything gets written. It's retrieval with a lookup table rather than vectors, but the principle is the same: check the library first. For any of these tools, the useful question is whether an answer can be traced back to its source.
Why not just paste everything into the prompt?
Modern windows are large. OpenAI's documentation lists GPT-5's context window at 400,000 tokens, roughly 300,000 words, and Google's Gemini models have advertised windows of a million tokens or more. Big, and still not big enough to change the math.
A whole document collection outruns a single window: a company wiki or a decade of support tickets may not fit. Cost scales with the tokens you ship, too. Retrieving the ten relevant pages beats shipping ten thousand every time someone asks a question, and the answer tends to come out more focused, since the model reads selected passages instead of a wall of text.
When RAG is the right tool
RAG fits answers that must come from a specific collection of text: policies, manuals, research libraries, help centers, anything that changes without warning and needs a source you can check. Plain generation fits tasks where training knowledge is enough: writing, brainstorming, summarizing what's in front of the model.
RAG has failure modes worth knowing. Retrieval can miss the passage that matters or grab a lookalike, and the model can still misread what it retrieved. Answers with citations are the practical check: when a RAG system points to the page it answered from, verifying takes seconds, and when it doesn't, treat the answer like any other model output and spot-check it.
FAQ
What does RAG stand for?
Retrieval-Augmented Generation. The name comes from the 2020 research paper that introduced the design: retrieve relevant text first, generate the answer from it second.
What is the difference between RAG and fine-tuning?
RAG adds knowledge at question time: update the documents and the answers update, no retraining. Fine-tuning bakes patterns into the model's weights, which suits a consistent style or skill rather than fast-changing facts. Many production systems use both together.
What is a vector database?
A database built to store embeddings and return the ones closest to a query. In a RAG system it holds one embedding per document chunk and performs the meaning-based lookup between a question and the passages that answer it.
Does RAG stop AI from making things up?
No. RAG hands the model supporting material to answer from, which cuts down on answers invented from thin air, but it's not a guarantee of truth. The retrieved passage can be wrong or misread, so a cited answer is something you can check rather than something to take on faith.
Do I need to code to use RAG?
No. File uploads in ChatGPT, documents in a Claude Project, and notebooks in NotebookLM all run retrieval before the model answers. Code enters when you want to build and control your own pipeline.
Want to go further?
- What Is Context in AI? Why It Makes or Breaks Your Results covers what the model can see, the constraint RAG works around.
- How to Analyze Data with ChatGPT: Beginner's Guide shows the upload-and-ask workflow on a single file.
- What Is an AI Agent? Explained Simply covers the systems that often use RAG to ground their answers.