Skip to content
AskVault Docs
Try Free →

What is Retrieval-Augmented Generation (RAG)? A 2026 guide

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 12 min read

What does RAG mean?

RAG stands for Retrieval-Augmented Generation. It's the architecture pattern that lets large language models like GPT-5, Claude Sonnet 4.6, and Gemini 3 answer questions from a specific document corpus instead of their general training data. Facebook AI Research published the original RAG paper in May 2020 (Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks").

The core mechanic is simple. When a user asks a question, you don't pass the question directly to the LLM. You first retrieve the most relevant documents from your knowledge base using vector similarity search. Then you pass both the question and the retrieved evidence into the LLM, with an instruction like "answer the question using only the provided context." The LLM generates a response grounded in real evidence.

This targets two problems in production AI. The first is hallucinations: the model invents plausible-sounding wrong answers. The second is stale knowledge: the model only knows what was in its training corpus, frozen at a specific date. RAG addresses both. The model is told to answer only from what you've shown it, which reduces hallucination (Shuster et al., 2021, "Retrieval Augmentation Reduces Hallucination in Conversation") without eliminating it, and you can update the knowledge base instantly without retraining anything.

The 5 steps in a RAG pipeline

A production RAG pipeline runs five distinct steps every time a user asks a question. AskVault's implementation follows this same pattern, with a few production-hardening decisions described later in this guide.

  1. Chunk and embed your content. Split every document into chunks (smaller for dense reference content, larger for narrative). For each chunk, call an embedding model and store the resulting vector in a vector database.
  2. Embed the user's question. When a user asks "How do I reset my password?", you embed that question using the same model that embedded the chunks. The result is a high-dimensional vector representing the question's meaning.
  3. Retrieve top-K nearest chunks. Query the vector database for the K chunks whose embeddings are most similar to the question embedding under cosine similarity. This returns the chunks most semantically relevant to the question. Not just keyword matches, but conceptually-related content.
  4. Build a prompt with the retrieved evidence. Construct a prompt that includes the question, the retrieved chunks (as labeled context), and an instruction to answer using only that context. The chunks act as the only source of truth for the LLM. Textbook RAG stops there and tells the model to reply "I don't know" whenever the context falls short. AskVault splits that case in two, because the two failures are not the same: when retrieval finds nothing, strict retrieval (on by default) skips the model entirely and returns a plain "I don't have that information in my knowledge base"; when retrieval finds material that is relevant but incomplete, the agent answers from what that material does cover. Grounding is unchanged in both cases: every specific number, price, date, name, policy or contact detail must come from the retrieved chunks.
  5. Generate and cite. Send the prompt to the LLM, then return the answer together with the sources it was grounded in so the user can check it. AskVault returns one list of sources per answer, each entry naming the source document (with its page URL for web pages), not a citation attached to each individual sentence. The source list is shown by default and can be turned off per workspace.

End-to-end response time depends mostly on the chosen model: generation is bound by its first-token latency plus per-token decode rate, and retrieval adds its own overhead.

Why RAG beats fine-tuning for business AI

Fine-tuning is the alternative to RAG. You train a custom LLM on your data so the knowledge is baked into the model's weights. It sounds attractive. Your model "knows" your business directly. For answering questions from your own content, RAG usually wins on three dimensions.

Cost. Fine-tuning means paying for a training run every time the knowledge changes. RAG pays only for indexing the changed documents and for retrieval at question time.

Update speed. Adding a new product or policy to a fine-tuned model requires another training run. In a RAG system you re-index the new document, and the next query can retrieve it.

Citation and auditability. Fine-tuned models can't tell you where a specific claim came from. The knowledge is encoded across the model's weights. RAG returns the source chunks the answer was drawn from, so you can verify and audit the output, which matters most in high-stakes areas like legal, medical and financial support.

Where fine-tuning still wins. Two narrow cases. First, when you need the model to learn a specialized output format or style that prompting can't reliably enforce. Second, when latency must be ultra-low (RAG adds a retrieval step).

What is a vector database?

A vector database stores numerical representations of text (called embeddings) and lets you search by semantic similarity rather than keywords. Traditional databases match strings exactly. Search for "password reset" and you only find documents containing those exact words. Vector databases match meaning instead. Search for "I forgot my login" and you find documents about password reset, account recovery, "can't sign in", and more.

The math: each embedding is a vector of floating-point numbers. Documents with similar meanings produce vectors that are mathematically close (small cosine distance). Search becomes "find the K vectors closest to this query vector". That's a fast operation when the vector database uses approximate-nearest-neighbor indexing.

Production vector database options in 2026 include Postgres with native vector extensions, Pinecone, Weaviate, Qdrant, and Chroma. Each has trade-offs around price, ops overhead, and feature surface. Choosing the right vector-store pattern is one of the more consequential architecture decisions a RAG product makes.

Chunking: harder than it looks

Splitting documents into chunks is a common place for a RAG system to go wrong. Naive chunking (split at a fixed length, ignore document structure) produces three failure modes:

  • Context-breaking splits. A list of 7 password-reset steps gets split across 2 chunks, so the agent retrieves steps 1-4 but misses 5-7 and answers incompletely.
  • Hierarchical signal loss. A page titled "Returns Policy" with sub-sections "Refunds", "Exchanges", and "Damaged Items" gets chunked into 3 pieces that each contain "Returns Policy" as a leading line but lose the relationship.
  • Vector noise. Very short chunks produce embeddings that match too liberally; very long chunks dilute the semantic signal.

Better chunking uses structure-aware splitting. Parse the document's headings, lists, and tables. Split on natural boundaries (paragraph breaks, list-item boundaries, section ends). Keep parent-section context in each chunk via a header prefix, and overlap adjacent chunks slightly for context continuity.

AskVault's chunker treats Markdown tables as a unit. When a table is too large for one chunk, it is split between rows and the header row is repeated in every piece, so no chunk is a headerless row fragment.

Surviving an embedding-provider outage

The embedding model is a dependency every RAG system leans on, and embedding providers do have outages. A pipeline that can only retrieve by vector similarity goes completely dark when the embedding API is down: it cannot embed the incoming question, so it cannot search at all. This is a genuine single-point-of-failure.

AskVault's hybrid retrieval has a path that does not depend on embeddings. Alongside meaning-based search it matches exact keywords across your whole workspace. If the embedding provider is unavailable, AskVault answers from keyword search instead, degraded but online, rather than returning an error. Indexing new content still requires the embedding provider; the fallback covers live chat, which is where an outage would otherwise be most visible to your customers. See how vector databases work for more on hybrid search.

Hybrid retrieval: vector + keyword + rerank

Pure vector search produces excellent semantic matches but misses exact-name retrieval. If a customer asks "How do I configure the API_TIMEOUT_SECONDS environment variable?", vector search might return a chunk about API configuration but miss the chunk that literally lists API_TIMEOUT_SECONDS because the question's vector emphasizes "configure" and "environment variable" semantically over the exact string.

The fix is hybrid retrieval. A common way to build it is to run vector search AND keyword search (BM25) in parallel, then merge the results with a technique such as Reciprocal Rank Fusion. The vector search catches semantic matches; the keyword search catches exact-name retrieval. Combined, hybrid retrieval catches matches that either lane alone would miss.

A second technique stacks on top: reranking. Take the top-K chunks from hybrid retrieval, then run them through a small cross-encoder model that scores each chunk against the question more carefully than the initial retriever. Keep the best few. This adds modest latency but can lift answer quality measurably on complex queries.

AskVault turns on hybrid retrieval and cross-encoder reranking for every workspace by default, not as a higher-tier add-on. See cross-encoder reranking for how the rerank stage works and why it runs on every workspace.

Common RAG failure modes and how to debug them

Four failure modes to check when a RAG answer is wrong:

Failure 1: wrong chunks retrieved. The vector search returns chunks that don't actually contain the answer. Debug by inspecting the retrieved sources in your dashboard. If the chunks are off-topic, your embedding model is mis-classifying the question. Try a different embedding model or improve query rewriting. If the chunks are on-topic but the answer is still wrong, the problem is in generation, not retrieval.

Failure 2: right chunks, LLM ignores them. The LLM has the evidence but generates from its training data anyway. This happens with weaker models (cheaper tiers) when the question is one the model "knows" the answer to. Fix: tighten the system prompt ("answer ONLY from the provided context, never from training data"), turn on strictness mode, or move up to a stronger model.

Failure 3: right chunks, right LLM behavior, wrong format. The answer is correct but the agent doesn't cite sources, formats badly, or rambles. Fix: improve the prompt's output-format instructions and add few-shot examples of the response shape you want.

Failure 4: indexing skipped the relevant content. The chunks containing the answer aren't in the vector database in the first place. The crawler missed them (JS-rendered content), the parser failed (image-only PDF), or the chunker truncated them (huge document). Fix: check the per-document indexing status in the dashboard, then recrawl or re-upload with the right settings.

In AskVault, each answer comes back with the sources that were retrieved for it (shown by default), and the Knowledge shows per-document indexing status. When auto-train is on for a workspace, the knowledge.gap_detected webhook fires once for each new clustered gap suggestion, not for every low-confidence answer.

When NOT to use RAG

RAG fits many business AI uses, but not all of them. Four cases where you should pick something else:

  • Deep reasoning over very long context (analyze this 500-page legal contract end-to-end). You want long-context single-shot inference with a model whose context window holds the whole document, not retrieval.
  • Multi-modal queries on a single document (find every chart in this PDF showing year-over-year decline). You want a vision-capable LLM with the full document, not chunk retrieval.
  • Ultra-low latency requirements. Retrieval adds measurable overhead. Use a lightweight rule-based or fine-tuned classifier instead.
  • Pure creative generation (write a poem about my company values). There's no factual grounding to do, so retrieval is wasted overhead.

For B2B customer support, internal helpdesks, technical documentation, e-commerce product Q&A, and API support, RAG is a strong fit. That's what AskVault is built for.

How AskVault implements RAG

AskVault's RAG implementation is engineered for production B2B workloads:

  • A crawler that handles JavaScript-rendered sites out of the box, so your modern marketing site, docs site, or single-page app is indexable without per-host configuration
  • Keyword search as a safety net that keeps chat answering even if the embedding provider has an outage
  • Managed database storage from a provider that holds a SOC 2 Type II report
  • Per-workspace data isolation so multi-tenant customers can't see each other's content
  • Hybrid retrieval that understands meaning and still catches exact names, with cross-encoder reranking on every workspace by default

AskVault answers are grounded in your indexed content, and by default each grounded answer is shown with the list of sources it was drawn from. That reduces hallucinations rather than eliminating them: the model is instructed to answer only from chunks retrieved from your knowledge base, and when retrieval finds nothing, strict retrieval returns a plain "I don't have that information" instead of calling the model at all.

FAQ

What does RAG stand for?

RAG stands for Retrieval-Augmented Generation. It's an AI architecture that combines vector search (retrieval) with large language models (generation) so the AI answers from a specific document corpus instead of its general training data.

How does RAG prevent hallucinations?

RAG retrieves relevant document chunks before generation, then instructs the LLM to answer only from those chunks. Grounding the model in retrieved evidence makes invented facts much less likely than with an ungrounded model, though no LLM system can rule them out entirely. If the answer isn't in the chunks, a well-tuned RAG system refuses to answer rather than guessing.

What is the difference between RAG and fine-tuning?

Fine-tuning bakes new knowledge into the model's weights, so every update means another training run, and the model cannot point to where an answer came from. RAG keeps knowledge external: you update it by re-indexing a document, and each answer can be returned with the sources it was drawn from. For answering questions from your own content, RAG is usually the better fit.

What is a vector database?

A vector database stores numerical representations (embeddings) of text and lets you search by semantic similarity rather than keywords. AskVault stores your content with a managed database provider that holds a SOC 2 Type II report.

How much does running a RAG system cost?

It depends on the workload. AskVault's plans are Free, Starter ($39/mo), Growth ($99/mo) and Business ($249/mo), each with a fixed monthly message-credit allowance and an indexed-content cap. See the pricing page for the per-tier matrix.

Can RAG work with private/sensitive data?

Yes, with a clear picture of where the data goes. AskVault uses hosted AI providers, so your content is sent to one whenever an AI step runs. For example: when you add a website, page text goes to a hosted model that summarises your business for setup; your content goes to a hosted embedding provider to create its search vectors; each answer sends the question, the retrieved passages, the recent turns of that conversation and your agent's instructions to a hosted model provider; and AI features you use in the dashboard, such as suggested replies, translation or generated Q&A, send the conversation or content they work on to a hosted model too. The vectors are stored in AskVault's database, scoped to your workspace. In a self-built RAG system you can go further and run both the embedding and the answer steps on self-hosted open-source models, so no data leaves your infrastructure.

Was this page helpful?