Large Language Models like GPT-4 and Claude are remarkable — but they have a fundamental limitation: everything they know was frozen at training time. Ask them about last week's news, your company's internal documents, or a product released yesterday and they either hallucinate an answer or admit they don't know. Retrieval-Augmented Generation (RAG) solves this by giving an LLM the ability to look things up before it answers — combining the reasoning power of a language model with the accuracy of a live knowledge base.
What is Retrieval-Augmented Generation?
RAG is an AI architecture pattern that augments a language model's response by first retrieving relevant information from an external knowledge source and injecting it into the prompt as context. The model then generates its answer grounded in that retrieved content rather than relying solely on what it learned during training.
The term was introduced in a 2020 paper by researchers at Facebook AI Research (now Meta AI). The core insight was simple but powerful: instead of trying to bake all knowledge into a model's parameters (which is expensive, inflexible, and quickly goes stale), let the model retrieve what it needs at query time from a separate, updatable knowledge store.
One-line definition
RAG = Retrieval + Generation. Find the relevant facts first, then generate an answer using those facts as context.
Why Do We Need RAG?
Language models have three limitations that RAG directly addresses:
Knowledge cutoff
The model only knows what was in its training data. Events, documents, or data after the cutoff date are invisible to it. A model trained in early 2024 knows nothing about what happened in late 2024 or 2025.
Hallucination
When an LLM doesn't know something, it often invents a plausible-sounding answer rather than admitting uncertainty. In business contexts — legal, medical, financial — a confident wrong answer is worse than no answer.
Private knowledge
Your company's internal documentation, product manuals, customer records, and proprietary research were never in the model's training data and never should be. RAG lets you query private knowledge safely.
Fine-tuning is an alternative — you retrain the model on your specific data. But fine-tuning is expensive, slow, requires significant ML expertise, and still doesn't solve the staleness problem because it too produces a static, frozen model. RAG is more practical for most use cases: the knowledge base can be updated in minutes without retraining anything.
How RAG Works — The Architecture
RAG has two distinct phases: an offline indexing phase (done once, or whenever your knowledge base updates) and an online retrieval-generation phase (done for every user query).
RAG Architecture
Phase 1 — Indexing (Offline)
Raw Documents
PDFs, docs, web pages, databases
Text Chunking
Split into ~500 token chunks
Embedding Model
Convert chunks to vectors
Vector Database
FAISS / Pinecone / Chroma
Phase 2 — Retrieval + Generation (Online, per query)
User Query
"What is our refund policy?"
Embed Query
Same embedding model as indexing
Vector Search
Find top-k similar chunks
Prompt Construction
Query + retrieved chunks
LLM Generation
Answer grounded in context
Step-by-step explanation
Document ingestion
You feed your knowledge base into the system — PDFs, Word documents, web pages, database tables, Notion pages, anything. These are the documents the model will be able to reference.
Chunking
Each document is split into smaller chunks, typically 200–500 tokens each. This is necessary because you can't put an entire 50-page PDF into a prompt. Chunk size is a tunable parameter — too small and chunks lose context; too large and retrieval becomes imprecise.
Embedding
Each chunk is converted into a vector (a list of numbers) using an embedding model. Semantically similar text produces similar vectors. This is what enables searching by meaning rather than keyword matching.
Vector storage
All chunk vectors are stored in a vector database (FAISS, Pinecone, Chroma, Weaviate). Think of it as a search index where you search by meaning rather than text.
Query embedding
When a user asks a question, that question is embedded using the same model, producing a query vector.
Similarity search
The query vector is compared against all chunk vectors using cosine similarity or dot product. The top-k most similar chunks are retrieved — these are the passages most likely to contain the answer.
Prompt construction
A prompt is assembled: the original user question + the retrieved chunks as context. The LLM is instructed to answer only based on the provided context.
Generation
The LLM generates a response grounded in the retrieved context. Because the relevant facts are in the prompt, the model doesn't need to rely on (potentially outdated or wrong) parametric memory.
Python Implementation — RAG from Scratch
Let's build a working RAG pipeline step by step using LangChain, FAISS, and OpenAI. We'll build a system that answers questions about a custom document.
Step 1 — Install dependencies
pip install langchain langchain-openai faiss-cpu tiktoken openai
Step 2 — Prepare and chunk documents
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Your source documents — could be PDFs, web pages, database exports
documents = [
"""
ML Academy Refund Policy:
All premium courses are eligible for a full refund within 30 days of purchase.
To request a refund, email support@mlacademy.com with your order ID.
Refunds are processed within 5-7 business days.
After 30 days, only partial refunds of 50% are available up to 60 days.
After 60 days, no refunds are issued.
""",
"""
ML Academy Course Catalogue:
We offer courses in Machine Learning, Deep Learning, NLP, and MLOps.
The Machine Learning Fundamentals course runs for 12 weeks.
The Deep Learning Specialisation runs for 16 weeks.
All courses include live mentoring sessions every Saturday at 10 AM IST.
Certificate of completion is issued within 7 days of finishing the course.
""",
]
# Split documents into chunks
splitter = RecursiveCharacterTextSplitter(
chunk_size=200, # characters per chunk
chunk_overlap=40, # overlap between consecutive chunks to preserve context
)
chunks = splitter.create_documents(documents)
print(f"Total chunks created: {len(chunks)}")
for i, chunk in enumerate(chunks):
print(f"\nChunk {i+1} ({len(chunk.page_content)} chars):")
print(chunk.page_content[:100], "...")Output
Total chunks created: 6 Chunk 1 (198 chars): ML Academy Refund Policy: All premium courses are eligible for a full refund within 30 days of purchase. To request a refund, email support@mlacademy.com ... Chunk 2 (156 chars): To request a refund, email support@mlacademy.com with your order ID. Refunds are processed within 5-7 business days. After 30 days, only partial ...
Step 3 — Create embeddings and vector store
import os
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
# Embedding model converts text to vectors
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
# Build FAISS vector store from chunks
vectorstore = FAISS.from_documents(chunks, embedding_model)
# Save to disk so you don't rebuild every time
vectorstore.save_local("my_knowledge_base")
print("Vector store created and saved.")
print(f"Indexed {vectorstore.index.ntotal} vectors")
# Test similarity search
query = "How many days do I have to request a refund?"
results = vectorstore.similarity_search(query, k=2)
print(f"\nTop 2 results for: '{query}'")
for i, doc in enumerate(results):
print(f"\nResult {i+1}:")
print(doc.page_content)Output
Vector store created and saved. Indexed 6 vectors Top 2 results for: 'How many days do I have to request a refund?' Result 1: ML Academy Refund Policy: All premium courses are eligible for a full refund within 30 days of purchase. To request a refund, email support@mlacademy.com with your order ID. Result 2: Refunds are processed within 5-7 business days. After 30 days, only partial refunds of 50% are available up to 60 days. After 60 days, no refunds are issued.
Step 4 — Build the full RAG chain
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
# Load vector store from disk
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.load_local(
"my_knowledge_base",
embedding_model,
allow_dangerous_deserialization=True
)
# Create retriever — fetches top 3 most relevant chunks per query
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
# Prompt template — instructs the LLM to answer only from context
prompt_template = """You are a helpful assistant. Answer the user's question
using ONLY the information provided in the context below.
If the answer is not in the context, say "I don't have that information."
Context:
{context}
Question: {question}
Answer:"""
prompt = PromptTemplate(
template=prompt_template,
input_variables=["context", "question"]
)
# LLM
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
# Full RAG chain
rag_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff", # stuff = concatenate all retrieved chunks into one prompt
retriever=retriever,
chain_type_kwargs={"prompt": prompt},
return_source_documents=True
)
# Ask questions
questions = [
"What is the refund policy after 30 days?",
"How long does the Deep Learning course run?",
"What time are the mentoring sessions?",
"Do you offer refunds after 90 days?",
]
for q in questions:
result = rag_chain.invoke({"query": q})
print(f"Q: {q}")
print(f"A: {result['result']}")
print(f"Sources used: {len(result['source_documents'])} chunks")
print("-" * 60)Output
Q: What is the refund policy after 30 days? A: After 30 days, only a partial refund of 50% is available up to 60 days. After 60 days, no refunds are issued. Sources used: 3 chunks ------------------------------------------------------------ Q: How long does the Deep Learning course run? A: The Deep Learning Specialisation runs for 16 weeks. Sources used: 3 chunks ------------------------------------------------------------ Q: What time are the mentoring sessions? A: Live mentoring sessions are held every Saturday at 10 AM IST. Sources used: 3 chunks ------------------------------------------------------------ Q: Do you offer refunds after 90 days? A: I don't have that information. Sources used: 3 chunks ------------------------------------------------------------
Notice the last answer
The model correctly says "I don't have that information" for the 90-day refund question — because that scenario isn't in the documents. This is RAG preventing hallucination: when the answer isn't in the retrieved context, the model admits it rather than inventing something.
Real-World Examples of RAG in Production
A company indexes its entire help documentation, product manuals, and FAQ database. When a customer asks 'How do I reset my password?', the bot retrieves the relevant help article and generates a precise, step-by-step answer — always up to date because the index updates whenever docs change.
Law firms index thousands of case files and regulatory documents. Associates ask 'Find precedents where GDPR Article 17 was applied in healthcare' and the RAG system retrieves the 5 most relevant cases for review — something that would take hours of manual search.
Hospitals build RAG systems over peer-reviewed clinical guidelines. When a clinician asks about drug interactions or treatment protocols, the system retrieves the specific guideline sections and generates a summary with citations — grounded in verified medical literature.
Enterprises index Confluence pages, Slack threads, engineering RFCs, and HR policies. Employees ask 'What is our parental leave policy in Germany?' and get a precise answer from the actual HR document — not a guess from a generic model.
When to Use RAG — Practical Decision Guide
RAG is not the right solution for every problem. Here is a clear decision framework:
| Scenario | Use RAG? | Why |
|---|---|---|
| Your knowledge base updates frequently (daily/weekly) | ✅ Yes | RAG indexes update cheaply. Fine-tuning can't keep pace. |
| You need answers grounded in specific private documents | ✅ Yes | RAG retrieves from your exact source material with citations. |
| You need to prevent hallucinations on factual queries | ✅ Yes | Grounding in retrieved context dramatically reduces confabulation. |
| You want to cite sources for every answer | ✅ Yes | RAG knows which chunks it retrieved — citation is straightforward. |
| You need the model to change its tone/writing style | ❌ Use Fine-tuning | RAG adds external facts; it doesn't change how the model writes. |
| You need to teach the model new tasks it can't do today | ❌ Use Fine-tuning | RAG provides context for retrieval; it doesn't teach new capabilities. |
| Your question requires reasoning over all documents at once | ⚠️ With care | RAG retrieves top-k chunks. Global reasoning across a corpus is harder. |
| Low-latency response is critical (under 100ms) | ⚠️ With care | Retrieval adds latency. Cache common queries to mitigate. |
Advantages of RAG
Always up to date
Update the index, not the model. New information is available in the next query without any retraining.
Reduces hallucination
When the answer is grounded in retrieved text, the model is far less likely to fabricate. The context is in the prompt.
Transparent and citable
You always know which source chunks were retrieved. Every answer can be traced back to a specific document section.
Cost efficient
No GPU clusters needed. Adding new knowledge means updating a document and re-indexing — minutes of work, not weeks of training.
Works on private data
Your documents never need to leave your infrastructure. You can run the embedding model and LLM entirely on-premises.
Scales to large corpora
Approximate nearest-neighbour search (FAISS, ScaNN) retrieves from millions of chunks in milliseconds.
Disadvantages and Limitations
Retrieval quality bottleneck
If retrieval returns the wrong chunks, the LLM will generate a wrong answer confidently. Garbage in, garbage out — the model cannot compensate for a bad retrieval step.
Chunk size is tricky
Too small: chunks lack context and retrieval is imprecise. Too large: the prompt fills up and important details from other chunks get squeezed out. Requires tuning per use case.
Latency overhead
Every query now involves an embedding step plus a vector search before the LLM call. For real-time applications, this added latency must be managed with caching and async retrieval.
Multi-hop reasoning is hard
If answering a question requires combining information from 5 different document sections, standard RAG (which retrieves a fixed k chunks) may miss some. Multi-hop RAG extensions address this but add complexity.
No persistent memory
RAG retrieves from static documents. It has no memory of previous conversations unless you explicitly add conversation history to the prompt.
Requires embedding consistency
If you change your embedding model, you must re-embed and re-index your entire knowledge base. Mixing embeddings from different models in the same index breaks similarity search.
When NOT to Use RAG
RAG is powerful, but it is the wrong tool in these specific situations:
When the task is purely generative with no factual grounding required
Writing a poem, generating creative fiction, brainstorming taglines — none of these benefit from retrieval. RAG adds latency and complexity without adding value when the task doesn't require factual accuracy.
When you need to change the model's core behaviour or capabilities
If your LLM can't summarise in bullet points, doesn't follow a specific output format, or lacks a domain-specific style — these are training problems, not retrieval problems. Fine-tuning changes how the model behaves; RAG only changes what it knows.
When your knowledge base is tiny and static
If you have 10 documents that never change, you can fit them all directly into a system prompt. RAG introduces indexing infrastructure, embedding costs, and retrieval complexity that simply isn't worth it for a small, unchanging corpus.
When you need precise mathematical or logical reasoning over data
RAG retrieves relevant text passages — it doesn't execute code or do arithmetic. If someone asks 'What was our average revenue last quarter across 10,000 transactions?', you need a tool-augmented LLM with database access, not RAG.
When response latency must be under ~200ms
An embedding call + vector search + LLM generation typically takes 1–4 seconds end to end. For real-time autocomplete, live transcription, or instant UI responses, RAG's architecture is too slow without significant caching infrastructure.
RAG vs Fine-tuning — Quick Comparison
| Aspect | RAG | Fine-tuning |
|---|---|---|
| Knowledge updates | Index update — minutes | Retraining — days to weeks |
| Cost | Low — indexing is cheap | High — requires GPU training |
| Hallucination control | Strong — grounded in retrieved text | Moderate — baked into parameters |
| Citable sources | Yes — you know which chunks were used | No — knowledge is in weights |
| Behaviour change | No — only knowledge changes | Yes — model behaviour can be reshaped |
| Private data safety | Documents stay in your store | Data is baked into model weights |
| Best for | Dynamic, factual Q&A over documents | Style, format, domain-specific tasks |
Key Takeaways
RAG gives LLMs access to external, up-to-date knowledge without retraining — by retrieving relevant text and injecting it into the prompt before generation.
The two phases are: offline indexing (embed and store your documents in a vector database) and online retrieval (embed the query, find similar chunks, generate a grounded answer).
RAG is the right choice when you need accurate, citable, up-to-date answers from a specific knowledge base. It is not the right choice for creative tasks, behaviour change, or sub-200ms latency requirements.
The biggest failure mode in RAG is not the LLM — it is retrieval quality. If the wrong chunks are retrieved, the best LLM in the world will give a wrong answer. Invest in chunking strategy and retrieval evaluation.
RAG and fine-tuning are complementary, not competing. Many production systems use both: fine-tune for style and behaviour, add RAG for up-to-date factual grounding.