What is RAG in AI? An open-book exam for your chatbot
RAG in plain English: how retrieval-augmented generation lets AI answer from your own documents with sources, what it costs, and when you do not need it.
RAG, short for retrieval-augmented generation, is a way of making an AI model answer from your own documents instead of from memory. Before it answers, the system searches your files for the passages most relevant to the question and hands them to the model along with it, so the answer is based on your information and can point to the page it came from. Think of it as turning a closed-book exam into an open-book one.
- RAG means look it up first, then answer. The model writes from passages found in your documents, and can show its source.
- It fixes the two big weaknesses of a plain chatbot: it knows nothing about your business, and it will sometimes make things up.
- It reduces made-up answers but does not eliminate them, so a good system cites sources and says "I don't know" when it should.
- Indexing thousands of pages costs pennies. The real work is tidying the documents and testing with real questions.
- If all your documents add up to fewer than about 500 pages, you may not need RAG at all.
What does RAG stand for?
The name comes from a 2020 research paper by Patrick Lewis and colleagues, and each word describes one part of the job:
- Retrieval: find the passages in your documents that relate to the question.
- Augmented: add those passages to the request sent to the AI model.
- Generation: the model writes the answer, using the passages as its material.
Nothing about the model itself changes. RAG changes what the model is given to read before it answers.
Why does a chatbot need an open book?
A language model on its own is like a student sitting a closed-book exam. It answers from what it absorbed in training: a great deal of general knowledge, a cut-off date, and nothing at all about your refund policy, your price list or last month's change to the staff rota. Ask it anyway, and it may write a confident, fluent answer that happens to be wrong.
That is not a theoretical risk. In 2024, a Canadian tribunal ordered Air Canada to refund a passenger the difference on his fare, plus interest and fees, after the airline's website chatbot told him he could claim a bereavement discount after booking. The airline's own policy said he could not. Air Canada argued the chatbot was responsible for its own actions; the tribunal disagreed, finding the airline "responsible for all the information on its website". Whatever was behind that chatbot, the lesson for every business is the same: answers must come from your real policy, and you own them.
RAG is how you hand the model the right pages before it answers.
How does RAG work, step by step?
Here is one question traced through a RAG assistant built on a fictional staff handbook.
Diagram
How a RAG assistant answers one question
Before anyone asks · Index
Your documents are split into short passages. Each passage is stored with a numeric fingerprint of its meaning, called an embedding.
1 · Ask
“How much notice do I need to give for a week’s holiday?”
2 · Retrieve
It finds the passages closest in meaning to the question:
- ▸ Staff handbook, 6.2 Annual leave
- ▸ Staff handbook, 6.4 Busy periods
- ▸ Rota policy, page 3
3 · Generate
The language model writes an answer using only those passages, under instructions not to add anything they do not say.
4 · Answer, with the source
Two weeks’ notice for a week off. In December it is four weeks, because that is a busy period.
SOURCE: Staff handbook, 6.2 and 6.4
In plain words:
- Index, once. Your documents are split into short passages. Each passage is turned into an embedding: a list of numbers that captures what it means. These are stored in a searchable index, and updated when a document changes.
- Retrieve. When someone asks a question, the question gets an embedding too, and the system finds the passages whose meaning is closest. That is why it can match "holiday" to a section headed "annual leave", which a keyword search would miss.
- Generate. The model receives the question, the passages and instructions to answer only from them.
- Answer, with the source. The reply links back to the document and section, so anyone can check it.
The search step is where most of the quality comes from, and it can be improved a lot. Anthropic reported in 2024 that adding a short note of context to each passage and combining meaning-based search with ordinary keyword search cut failed retrievals by 49%, and by 67% once a reranking step re-sorted the results.
What is RAG used for?
Anywhere people ask questions whose answers are written down somewhere, but hard to find.
| Use | Who asks | What it reads | A typical question |
|---|---|---|---|
| Staff help desk | Employees | Handbook, HR policies, how-to guides | "How do I claim expenses for mileage?" |
| Customer help on a website | Customers | Product guides, returns and delivery policies | "Can I return a sale item?" |
| Product and spec lookup | Sales and support teams | Spec sheets, manuals, price lists | "What is the maximum load on the 3-metre model?" |
| Field and technical support | Engineers, technicians | Service manuals, past job notes | "What does error E4 mean on this boiler?" |
| Contract and policy search | Managers | Contracts, insurance policies, leases | "What notice period does our cleaning contract have?" |
| A personal research library | Individuals | Your own notes, papers and saved articles | "What did I write about pricing last spring?" |
A RAG chatbot is simply one of these with a chat window on the front. It can sit on your website, inside Slack or Microsoft Teams, or in your help desk. If you want a customer-facing chatbot more generally, see our AI chatbots and assistants service.
Is ChatGPT a RAG?
Not on its own. The model behind ChatGPT answers from what it learned in training. But ChatGPT does use retrieval in some features: when it searches the web, and when you upload files, where OpenAI's help pages for ChatGPT Enterprise describe it adding relevant text to the prompt and using a search system to find more. That is RAG in all but name.
The difference with a RAG system built for a business is control. It is connected to your live documents rather than files someone uploaded once, it follows your rules about who may see what, it cites its sources every time, and it is tested against the questions your staff and customers really ask.
RAG vs fine-tuning vs pasting it all in
"Can we train the AI on our data?" is usually the first question. There are three ways to give a model your knowledge, and RAG is the right one more often than people expect.
| Option | What it means | Best for | Watch out for |
|---|---|---|---|
| Paste it all in | Put every document into the prompt each time | Small collections, under about 500 pages | Cost per question grows with the pile, and sources are not shown by default |
| RAG | Look up the relevant passages first, then answer | Large or changing collections, answers that need a source, different permissions for different people | Messy documents and missed passages |
| Fine-tuning | Train the model further on your own examples | Teaching a style, a format or a narrow repeated task | A poor fit for facts that change, because every change means training again |
The first row is worth taking seriously. Anthropic's own guidance is that if your knowledge base is smaller than 200,000 tokens, roughly 500 pages, you can include all of it in the prompt and skip RAG, with prompt caching keeping it fast and affordable.
What about MCP?
MCP, the Model Context Protocol, comes up next to RAG a lot, but it solves a different problem. MCP is an open standard for connecting AI applications to tools and data, which its documentation describes as being like a USB-C port for AI. RAG is about finding the right passage once the AI can reach your data. They often work together: MCP connects an assistant to your shared drive, and RAG finds the right page inside it. If you want AI connected to systems you already run, that is our AI integration work.
What does a RAG system cost to run?
Less than most people expect for the AI itself. Here is a worked example for a business with 5,000 pages of manuals, policies and guides, at prices checked in October 2026. A token is the unit AI models are billed in, roughly three-quarters of an English word.
| Item | Assumption | Cost |
|---|---|---|
| Index the whole library once | 5,000 pages × 500 words ≈ 3.3 million tokens, at OpenAI's $0.02 per million tokens | About 7 cents |
| Answer one question, larger model | 4,000 tokens in (question, instructions, five passages) and 300 out, on Claude Sonnet 5.5 at $2 in and $10 out per million | About 1.1 cents |
| Answer one question, smaller model | The same, on Claude Haiku 5.5 at $0.10 in and $0.50 out per million | About 0.06 cents |
| 1,000 questions a month | On the larger model | About $11 |
On top of that come hosting for the index and the app, which depends on where it runs, and the part no price list shows: people's time to gather the right documents, remove the old versions, set permissions and write the test questions. That preparation decides whether the assistant is any good.
Where RAG goes wrong
RAG is not magic, and it helps to know the failure points before you build one.
- Messy sources. Three versions of the same policy, old price lists and duplicates all get retrieved, and the model cannot tell which is current. Scanned PDFs are a trap too: on ChatGPT plans other than Enterprise, for example, OpenAI says document retrieval uses only the digital text in a file and discards images, so a scanned page with no text layer is invisible.
- The search misses. The answer is in your documents, but the retrieval step does not find it. That is why the techniques above, and testing, matter so much.
- It still makes things up, sometimes. Stanford researchers found in 2024 that legal research tools built with retrieval made fewer errors than a general chatbot, yet still gave incorrect information more than 17% of the time, and one tool more than 34%. Their conclusion was blunt: "even RAG systems are not hallucination-free".
- Questions about the whole pile. "How many of our contracts renew this year?" needs every contract counted, but RAG only reads a handful of passages per question. Questions like that need the data extracted into a table first, which is what AI document processing does.
- Permissions. If everything is indexed together, the assistant can quote the salary review to the wrong person. Access rules have to follow your existing groups from day one.
A well-built system answers these with source links, a refusal when nothing relevant is found, clean sources and a standing set of test questions that is re-run whenever anything changes.
Do you need a RAG chatbot? A quick checklist
You probably do if most of these are true:
- Your answers live in more than a few hundred pages, or change often.
- People ask the same questions again and again, by email, chat or across the desk.
- Answers need a source that someone can check.
- Different people should see different documents.
- Your information cannot go into a public chatbot.
You probably do not, yet, if:
- Everything fits in a few dozen pages. Paste it into a business AI plan's project space, or use a simple chatbot.
- The questions are about numbers in a database. That is a reporting job, not a reading one.
- Nobody keeps the documents up to date. Fix that first, because RAG will faithfully repeat whatever is out of date.
What we would do
Start with one collection that causes the most repeat questions, usually the staff handbook or the customer help pages. Collect fifty real questions from your inbox and your team, write down the correct answer to each, build the assistant, and score it against that list before anyone relies on it. Once it passes, add the next collection. The same assistant can later power an agent that looks things up as part of a job, which we cover in our AI agent examples.
Next Step
Get answers from your own documents
A private RAG assistant that cites its sources and keeps your information yours.
Frequently Asked Questions
What is RAG in simple terms?
RAG, short for retrieval-augmented generation, means the AI looks things up before it answers. It searches your documents for the passages most relevant to the question, then writes its answer from those passages, so it can tell you where the answer came from.
Is ChatGPT a RAG model?
Not on its own. The underlying model answers from what it learned in training. ChatGPT does use retrieval in some features, such as searching the web or searching files you upload, which is the same idea. A business RAG system goes further: it is connected to your own documents, respects who may see what, and cites its sources every time.
What is the difference between RAG and an LLM?
An LLM (large language model) is the part that understands questions and writes answers. RAG is a method wrapped around it: a search step that finds relevant passages in your documents and hands them to the LLM, so its answer is based on your information rather than its memory.
What is the difference between RAG and MCP?
RAG is about finding the right passage to answer from. MCP, the Model Context Protocol, is an open standard for connecting AI applications to tools and data sources, which its documentation compares to a USB-C port. They work together: an MCP connection can be how an assistant reaches your drive or database, and RAG is how it finds the right page inside.
Does RAG stop AI hallucinations?
It reduces them but does not stop them. A 2024 Stanford study found that legal research tools built with retrieval made fewer errors than a general chatbot, but still gave incorrect information in more than 17% of test questions. Good RAG systems add source links, refuse to answer when nothing relevant is found, and are tested against real questions.
What are the steps of RAG?
They are usually described as five: load your documents, split them into passages, turn each passage into an embedding and store it, retrieve the passages closest to each question, and generate an answer from them. The first three happen once, and again when documents change; the last two happen every time someone asks.
Next Action
Want help implementing this in your business?
Tamkoz builds AI phone receptionists, bespoke chatbots, and business automations. All work is quoted in writing before starting.