All articles
165 articles · updated weekly See our Tools
All articles
Tutorials

What Is RAG: How to Connect an AI to Your Own Data

RAG (Retrieval-Augmented Generation) fetches the right chunks of your documents at query time and hands them to the LLM. The pipeline, hybrid search, and limits.

What Is RAG: How to Connect an AI to Your Own Data
COVER · Tutorials

You wire a chatbot into the company intranet, ask "what's the travel reimbursement policy?", and the model answers with absolute confidence — quoting an amount, a deadline, and even a form that doesn't exist. The text reads perfectly. It is entirely wrong. The problem isn't that the model is "dumb": it's that it has never seen your internal documentation. An LLM only knows what was in its training data, which froze on some past date and never included the PDF that HR updated last week.

This article explains what RAG (Retrieval-Augmented Generation) is, how it closes that gap by connecting the model to your own data, how the pipeline used in production today works, and when it is — or isn't — the right tool. If vectors are new to you, the post on what embeddings are is the foundation; here the focus is the architecture around them.

The problem: the model doesn't know your data

An LLM is, in practice, a compressed memory of what it read during training. Two practical consequences follow. First, it has a knowledge cutoff: anything that happened after, or anything private to your company, simply doesn't exist for it. Second, when you ask about something it doesn't know, it rarely says "I don't know" — it fills the gap with the statistically most likely continuation. That's hallucination: plausible text, invented facts.

If you want to understand why the model generates so confidently even when it's wrong, it's worth reading how LLMs generate responses — it helps calibrate how much to trust the raw output. If the whole subject is new, start with what generative AI is.

The common reaction is "so we need to train the model on our data." Almost always that's the wrong conclusion, and I'll come back to it. The cheapest, most direct solution in most cases is different: don't change the model, change what you put in the question.

What RAG is, in one sentence

RAG means giving the model, at query time, the relevant chunks of your data — pasted into the prompt as context — before it answers. Instead of trusting its training memory, the model reads the material you just handed it and answers based on that.

The honest analogy: it's the difference between a student taking an exam from memory and taking the same exam open-book. The student is still the same person; what changes is what's on the desk when they answer. RAG is the technique of putting the right page on the desk, automatically, for every question.

The "retrieval" is the part that fetches the right passage. The "augmented generation" is generating the answer with that passage injected into the prompt. None of this rewrites the model — it stays the same weights file as always.

The name comes from the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues at Facebook AI Research, presented at NeurIPS 2020. Combining search with generation is older than that, but this work named the technique and showed that the model produced more specific, factual text when it consulted an external corpus (Wikipedia, in the experiment) than when it relied on its own weights alone.

What it's for: real use cases

RAG fits whenever the right answer lives in a set of documents the model never saw in training and that changes over time. The cases that actually show up in companies:

  • Internal policy and HR. The opening example: reimbursement, vacation, benefits. The document changes every year and the question is always the same.
  • Support with a knowledge base. The agent (human or bot) asks in natural language and gets the answer plus the link to the help-center article that backs it.
  • Contracts and legal. "What's the termination penalty in the contract with supplier X?" The value of RAG here is showing the exact clause, not just the answer.
  • Technical documentation. Runbooks, ADRs, the engineering wiki. A new hire asks "how do I deploy the payments service?" and gets the passage from the current runbook, not the one from two years ago.

In all of them, the gain isn't only less hallucination: it's being able to cite the source. An answer that comes with "per section 4.2 of the travel policy" can be checked in ten seconds.

The pipeline, from ingest to answer

It splits into two phases. One runs once (or whenever the data changes), the other runs on every question.

Indexing phase (offline):

documents
  -> chunk      (split each doc into pieces of ~200-800 tokens)
  -> embed      (each chunk becomes a vector of numbers)
  -> store      (save vector + text in a vector DB)

Query phase (every question):

user question
  -> embed                 (the question becomes a vector, same space)
  -> similarity search     (find the top-k closest chunks)
  -> stuff into prompt      (paste chunks + question into one prompt)
  -> generate              (the LLM answers using that context)

The key point: the model never "memorizes" your data. On every question, the system fetches the relevant pieces and hands them in fresh in the prompt. Updated a document? Just re-index that document — no retraining anything.

Hybrid search and reranking: the production pipeline

The diagram above is tutorial RAG. In production, the search step is almost never vector similarity alone, because embeddings are great at synonyms and bad at exact terms. Ask about "error E-1042" and semantic search brings back the reimbursement policy in general, not the FAQ line about that code. What finds error codes, product names, and clause numbers is classic keyword search: BM25.

Hence hybrid search: run both searches and fuse the results. The most common fusion method is Reciprocal Rank Fusion (RRF), from Cormack, Clarke, and Büttcher (SIGIR 2009), which Elasticsearch implements with a default constant k = 60 (Elastic docs). It ignores the scores, which come on incompatible scales, and only sums the inverse of the positions:

def rrf(rankings, k=60):
    """Reciprocal Rank Fusion: merges ordered lists of chunk ids."""
    scores = {}
    for ranking in rankings:
        for pos, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (k + pos)
    return sorted(scores, key=scores.get, reverse=True)

# question: "what does error E-1042 mean in reimbursement?"
bm25    = ["faq-errors#E-1042", "system-manual#codes", "reimbursement-policy#deadlines"]
vectors = ["reimbursement-policy#deadlines", "faq-errors#E-1042", "reimbursement-policy#limits"]

print(rrf([bm25, vectors]))
# ['faq-errors#E-1042', 'reimbursement-policy#deadlines', 'system-manual#codes', 'reimbursement-policy#limits']

I ran this snippet: the chunk that ranks well in both lists rises to the top, and what only one search found stays in the result, lower down.

After fusion comes reranking: a smaller model (a cross-encoder or a rerank API) reads question and chunk together and reorders, say, the first 150 candidates to keep the best 20. It's more expensive per pair than vector search, so it only runs over the short list. The gain is measurable: in Anthropic's Contextual Retrieval experiment (September 2024), the failure rate for retrieving the top 20 chunks was 5.7% with plain embeddings, dropped to 2.9% combining contextual embeddings with BM25, and reached 1.9% with reranking on top. These numbers come from the vendor's own benchmark, so treat them as an order of magnitude, not a guarantee for your data.

Embeddings and vector DBs, no mysticism

An embedding is a way to turn text into a list of numbers (a vector) such that texts with similar meaning land close together in that numeric space. "Reimbursement policy" and "how I get repaid for expenses" use different words but have close embeddings, because they mean almost the same thing. That's exactly what makes search work even when the user doesn't use the document's exact words. How those vectors are produced, and why the distance between them measures meaning, is in the post on embeddings; for RAG, you only need to know that question and chunks must go through the same embedding model.

A vector DB (Pinecone, Qdrant, pgvector, etc.) is a database optimized for one operation: given a query vector, quickly return the k closest vectors. "Closest" is usually measured by cosine similarity. For small volumes, you don't even need a dedicated database — an in-memory search does the job. A dedicated index starts to pay off around tens of thousands of chunks, when exhaustive search starts to weigh on latency; if you already run PostgreSQL, pgvector is usually the first step before a separate service.

Chunking: the decision that most affects quality

Here lives the part people underestimate. You don't embed the whole document — you embed pieces (chunks). And the size and the cut of those pieces decide what retrieval can find.

Chunks too large: the vector becomes an average of several topics, turns generic, and search drags in too much context that dilutes the signal. Chunks too small: you split an idea in half and the retrieved passage doesn't carry enough information to answer. Cutting in the middle of a sentence or a table is a classic source of bad answers. Good strategies respect the text's structure: split by section, by paragraph, or by heading, with a small overlap between chunks so you don't lose context at the boundary.

Before embedding, it's worth spending five minutes actually looking at the source text — analyzing term frequency helps you see the dominant vocabulary, decide chunk size, and spot noise (repeated headers, footer boilerplate) you don't want to index. It's not magic, just a cheap way to understand the material before you throw it all into the pipeline.

RAG or fine-tuning?

This is the most common confusion, so be blunt: the two solve different problems.

RAG is for facts and freshness — "what does our contract with client X say", "what's yesterday's release number". You want the model to cite specific, correct information that changes often. Fine-tuning is for style and format — making the model always answer in a specific tone, in strict JSON, or follow a classification pattern. Fine-tuning teaches behavior, not reliable new facts.

My opinion, after seeing this several times: most "we need fine-tuning" is actually a retrieval problem in disguise. Someone wants the model to know internal data, tries to teach it via fine-tuning, spends a lot, and the model keeps hallucinating because fine-tuning isn't a reliable way to inject facts. Start with RAG. Only consider fine-tuning when the problem is behavior, not knowledge.

The limits: retrieval is the ceiling

RAG isn't magic, and it has a hard limit: answer quality never exceeds retrieval quality. If search brought back the wrong chunk, the model will confidently answer from the wrong material. Garbage in, garbage out still holds — now applied to what you retrieve. If your documentation is ambiguous, outdated, or contradictory, RAG will reflect that faithfully.

The other limit is the context window: you can only inject a finite number of chunks into the prompt. A top-k that's too large not only costs more but can bury the useful passage in noise. And remember the model can still hallucinate even with good context — RAG reduces, doesn't eliminate. It's worth instructing the prompt to answer "I couldn't find this in the documents" when the context doesn't cover the question; the post on writing a good prompt shows how to write that kind of constraint without ambiguity.

The cheapest defense against the hallucination that remains is showing the source. Ask the model to cite the id of each chunk it used and display the original passage next to the answer. The user checks it in seconds, and you get a log of what retrieval brought back when an answer comes out wrong.

RAG or a giant context window?

With models accepting hundreds of thousands of tokens, the question comes up: can't you just paste everything into the prompt? Sometimes you can. Anthropic itself suggests that if the whole knowledge base is under 200,000 tokens (about 500 pages), it's worth trying to send all of it in the prompt with prompt caching before building a pipeline (Anthropic). Above that, or with many questions per minute, RAG wins again for three reasons: cost (you pay for input tokens on every question), latency, and attention. The Lost in the Middle study (Liu et al., 2023) showed that models make better use of information at the beginning or end of the context and degrade significantly when it sits in the middle. Long context and RAG don't compete: search picks what's worth putting in the window.

Frequently asked questions

Does RAG eliminate hallucination?

It reduces it a lot, doesn't eliminate it. By anchoring the answer in real passages, you remove the biggest source of invention — the model answering from memory about what it doesn't know. But if retrieval brings the wrong passage, or if the model extrapolates beyond what the context says, hallucination returns. So instruct the model to ground itself only in the context and admit when it can't find something.

Do I need a vector DB to start?

No. For a handful of documents, an in-memory similarity search works and is simpler to debug. A dedicated index (pgvector or a vector DB) starts to pay off when volume reaches tens of thousands of chunks and search latency matters. Start simple.

What's the ideal chunk size?

There's no universal number, but the 200 to 800 token range works for most text, with a small overlap. The right move is to respect the content's structure: split by section or paragraph instead of cutting by blind character count. Test with real questions and adjust.

Does RAG work with any model?

Yes — RAG is model-independent, because it only changes what goes into the prompt, not the model itself. Models with a larger context window can take in more chunks at once, which helps in context-heavy cases, but the pipeline is the same. The same goes for agents: in an AI agent, search becomes a tool the model calls when it needs it, instead of a fixed step before every answer.

Retrieval decides, not the model

RAG solves the most common problem in applied AI: the model doesn't know your private data or anything recent. Instead of rewriting the model, you fetch the right passages at query time and inject them into the prompt as context. The tutorial pipeline is chunk, embed, store, similarity search, and generate; the production one adds hybrid search with BM25 and a reranker before building the prompt. The things that decide the outcome are chunking and retrieval quality, not the model, and citing the source of every answer is what makes the system trustworthy. If your instinct was "we need fine-tuning," be suspicious: most of the time it's a retrieval problem, and starting with RAG is cheaper, faster, and easier to fix.

RD
Author
Rafael Duarte
Backend developer with a background in fintech and B2B SaaS — worked on teams that scaled APIs from zero to millions of requests. Carries enough production scars to hold strong opinions on tools, patterns and architecture decisions. Not an academic: he read the UUID RFC when he had to choose between v4 and v7 for a write-heavy table.
View profile