"RAG is to LLMs what Google is to a student — not smarter memory, but the ability to look up the right answer before speaking."
— Common framing in AI engineering circles, 2025–2026
Every business that has experimented with a large language model has hit the same wall: the model gives a confident, well-written answer that is factually wrong. It invents a product SKU that does not exist, cites a policy that changed six months ago, or describes a competitor's feature as if it were yours. This is called hallucination — and it is not a bug that future model versions will simply eliminate. It is a structural property of how LLMs work.
Retrieval-Augmented Generation (RAG) is the most widely deployed solution to this problem in production AI systems today. This guide explains what RAG is, how it works step by step, when to use it versus fine-tuning, which vector databases are in use in 2026, and how to estimate the cost of building a RAG system for an Indian business.
Why LLMs hallucinate — and why RAG fixes it
An LLM is trained on a snapshot of text — web pages, books, code, Wikipedia — up to a cutoff date. After training, its knowledge is frozen. It has no awareness of:
- Events that happened after its training cutoff
- Your company's proprietary documents, policies, and product catalogue
- Your customers' account records and order history
- Real-time prices, inventory, or live data of any kind
When asked about any of these, the model does not say "I don't know." It pattern-matches against what it was trained on and generates a plausible-sounding answer. This plausible-but-wrong behaviour is hallucination.
RAG changes the architecture. Before the LLM generates an answer, a retrieval step fetches the actual relevant documents from a knowledge base. Those documents are passed to the LLM as context — the model is then answering a question with the source material in front of it, not from memory alone. The result is dramatically more accurate, grounded, and citable.
How RAG works: the pipeline step by step
A RAG system has two distinct phases: an indexing phase (run once when the knowledge base is set up or updated) and a query phase (run on every user question).
Phase 1: Indexing your documents
Source documents — PDFs, Word files, web pages, database records, product catalogues, support tickets — are loaded and split into chunks of roughly 300–800 tokens each. Chunking strategy matters: too small and context is lost; too large and retrieval becomes imprecise. Good chunkers respect sentence and paragraph boundaries.
Each chunk is passed through an embedding model — a neural network that converts text into a dense numerical vector (typically 768–3072 dimensions). Chunks with similar meaning land close together in this vector space, regardless of the specific words used. OpenAI's text-embedding-3-large and Cohere's embed-v3 are the leading commercial choices; open-source alternatives like bge-large-en and multilingual-e5-large are strong for on-premise deployments.
The vectors — along with the original chunk text and metadata (document title, date, section, access permissions) — are stored in a vector database optimised for approximate nearest-neighbour (ANN) search. Popular choices include Pinecone, Weaviate, Chroma, Qdrant, and pgvector. The index is rebuilt or updated incrementally when documents change.
Phase 2: Answering a query
The user's question is run through the same embedding model used during indexing, producing a query vector that captures the question's semantic meaning.
The vector database performs a similarity search — finding the document chunks whose vectors are closest to the query vector. Typically the top 3–10 most relevant chunks are retrieved. This is where RAG's power lies: it finds semantically relevant passages even when the user's wording differs entirely from the document's.
The retrieved chunks are inserted into the LLM's prompt alongside the user's question: "Using the following excerpts from our documentation, answer the question: [question]. Context: [chunk 1] [chunk 2] [chunk 3]." The model now has ground truth in front of it.
The LLM reads the context and generates an answer citing the source material. A well-designed RAG system instructs the model to say "I don't find this in the provided documents" rather than guess when the retrieved chunks do not contain a relevant answer — preventing the original hallucination problem.
RAG vs fine-tuning: which approach is right?
Fine-tuning and RAG are often confused. They solve different problems and are frequently used together. This comparison clarifies the distinction:
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What it changes | What the model knows at query time | How the model thinks and writes |
| Knowledge update | Instant — update the vector DB | Requires full retraining cycle |
| Source citation | Yes — can return source documents | No — knowledge baked into weights |
| Handles private/confidential data | Yes — documents never leave your infrastructure | Risk of data leakage into model weights |
| Cost to set up | ₹1.5L–₹4L for a typical system | ₹5L–₹20L+ for quality fine-tuning |
| Best for | Q&A over documents, support bots, internal wikis | Custom writing style, domain-specific reasoning patterns |
The practical rule: if your goal is to get accurate answers from your company's data, use RAG. If your goal is to change how the model writes or reasons — making it sound like your brand voice, or teaching it an entirely new domain of logic — consider fine-tuning. Advanced production systems often combine both.
Advanced RAG patterns used in 2026
Basic RAG (embed → retrieve → generate) works well for clean document collections. Production systems typically add one or more of the following enhancements:
Re-ranking
The initial vector search retrieves candidates by approximate similarity. A second-pass re-ranker (typically a cross-encoder model like Cohere Rerank or a fine-tuned BERT) reads both the query and each candidate chunk together and produces a more precise relevance score. Re-ranking dramatically improves answer quality when the document collection is large or noisy.
Hybrid search
Combines dense vector search (semantic similarity) with sparse keyword search (BM25 / TF-IDF). Keyword search excels at exact matches — product codes, names, error messages — where semantic search can miss. Hybrid retrieval with a fusion step (Reciprocal Rank Fusion) outperforms either alone on most enterprise datasets.
HyDE (Hypothetical Document Embeddings)
Instead of embedding the user's question directly, the LLM first generates a hypothetical ideal answer to the question, then that hypothetical is embedded and used for retrieval. Because the hypothetical answer looks more like a document than a question, it retrieves better-matching chunks. Useful for complex questions where the query wording is very different from document phrasing.
Agentic RAG
The retrieval step is controlled by an AI agent that can issue multiple queries, decide when the context is sufficient, request different document sources, and iterate until it has enough information. This handles complex multi-hop questions that a single retrieval pass cannot answer. See our guide to AI agents for more on this pattern.
Vector databases: the right choice for your RAG system
The vector database is the backbone of any RAG system. The right choice depends on your scale, existing infrastructure, and deployment constraints:
| Database | Best for | Deployment | Cost (indicative) |
|---|---|---|---|
| Pinecone | Managed, production-ready, zero ops overhead | Cloud (SaaS) | Free tier → $70/month+ |
| Weaviate | Rich filtering, multi-modal, enterprise features | Cloud or self-hosted | Free tier → $25/month+ |
| Chroma | Local dev, small projects, prototyping | In-process / self-hosted | Open source (free) |
| Qdrant | High-performance, fine-grained filtering, data sovereignty | Self-hosted or cloud | Open source / $25/month cloud |
| pgvector | Already on PostgreSQL, simpler stack, <1M vectors | Your existing DB | No additional cost |
For most Indian SMEs and startups starting out, we recommend Chroma for prototyping and Qdrant or pgvector for production — both are open source, can be self-hosted on GCP or AWS, and keep data within Indian infrastructure. Pinecone is the easiest path to zero-ops production if managed cost is acceptable.
Real business use cases for RAG in 2026
Customer support knowledge base
The most common RAG deployment. Index your product documentation, FAQs, return policies, troubleshooting guides, and past resolved tickets. A support bot backed by this knowledge base answers routine queries with near-perfect accuracy, cites the relevant policy, and escalates to a human when the answer is not found. Typical outcome: 60–75% deflection rate from Level 1 support. The bot never invents a policy because it can only answer from indexed documents.
Internal employee assistant
Index HR policies, IT runbooks, compliance guidelines, internal SOPs, and project wikis. Employees ask questions in natural language and get instant, accurate answers with the source document linked. Cuts the back-and-forth emails asking "where is the leave policy?" or "what is the process for onboarding a new vendor?" Particularly valuable for companies with large, distributed teams and documentation that changes frequently.
Legal and compliance document Q&A
Index contracts, regulatory filings, vendor agreements, and compliance checklists. Lawyers and compliance officers can ask: "Does the MSA with Vendor X include a data residency clause?" and get a precise answer with the source clause. Reduces time spent manually searching through hundreds of pages of documents. RAG excels here because legal documents use precise, specific language that keyword search handles poorly.
E-commerce product discovery
Index the full product catalogue — names, descriptions, specifications, compatibility notes, customer reviews. A conversational search assistant answers queries like "What's the best laptop under ₹60,000 for video editing?" with specific product recommendations grounded in actual catalogue data. Unlike a traditional search filter, it understands intent and handles multi-criteria queries naturally.
Financial services client advisory
Index mutual fund factsheets, investment policy statements, regulatory disclosures, and market reports. A RAG-backed advisory assistant can answer client questions about fund performance, fee structures, and regulatory requirements — accurately, with citations — while staying strictly within the bounds of indexed, approved materials, reducing compliance risk.
The RAG tech stack iZenTech builds with
For RAG systems, our standard stack in 2026 is:
- Orchestration: LangChain or LlamaIndex — both provide ready-made abstractions for document loading, chunking, embedding, retrieval, and prompt assembly. LlamaIndex (now LlamaCloud) is particularly strong for document-heavy RAG; LangChain has a broader ecosystem for integrations.
- Embedding model: OpenAI text-embedding-3-large for English-primary applications; multilingual-e5-large or Cohere embed-multilingual-v3 for Indian-language support.
- Vector store: Qdrant (self-hosted on GCP) for production systems requiring data sovereignty; pgvector for systems already on PostgreSQL; Pinecone for fastest time-to-deploy.
- LLM: Claude (Anthropic) for its long context window (ideal for RAG with large chunks) and strong instruction following; GPT-4o for its tool-use capability in agentic RAG.
- Infrastructure: Google Cloud Run for the API layer; Cloud Storage for raw document hosting; BigQuery for analytics on query patterns and retrieval quality.
- Evaluation: RAGAS — the open-source RAG evaluation framework — for measuring faithfulness, answer relevancy, context precision, and context recall before going to production.
Cost to build a RAG system in India (2026 estimates)
RAG costs divide into two categories: build cost (one-time) and running cost (monthly).
Build cost
- Simple RAG bot (single document source, up to ~500 documents, standard chat UI): ₹1,50,000 – ₹3,00,000
- Production RAG system (multiple document sources, access controls, reranking, analytics, admin dashboard): ₹4,00,000 – ₹8,00,000
- Enterprise multi-source RAG (integrated with CRM, ticketing, ERP, role-based data access, custom UI): ₹8,00,000 – ₹18,00,000
Monthly running cost (SME volumes: ~5,000 queries/month)
- LLM API (Claude or GPT-4o): ₹6,000 – ₹20,000 depending on average context length
- Embedding API: ₹500 – ₹2,000
- Vector database (Qdrant cloud or Pinecone starter): ₹2,500 – ₹6,000
- Cloud Run hosting: ₹1,000 – ₹3,000
- Total: roughly ₹10,000 – ₹31,000/month at SME scale
Running costs scale primarily with query volume and context length, not with document count. A company that indexes 10,000 internal documents but handles 500 queries per month has very low running costs — the expensive part is the LLM call at query time, not storage.
Common RAG implementation mistakes to avoid
- Poor chunking strategy. Splitting documents by a fixed character count breaks sentences mid-thought. Use recursive text splitters that respect paragraph and sentence boundaries. For structured documents (PDFs with tables, HTML), parse the structure before chunking.
- No evaluation before launch. A RAG system that scores 70% on faithfulness (i.e. 30% of answers are partially hallucinated) will damage trust with users. Run RAGAS or a custom evaluation suite against a golden question set before deployment.
- Ignoring metadata filtering. When a user asks "what is our policy on remote work?", they mean the current policy — not a version from 2022. Metadata filters (date, document type, department) prevent the retrieval step from surfacing outdated or irrelevant chunks.
- Treating it as a one-time build. RAG systems need maintenance: document ingestion pipelines that run on schedule, monitoring for retrieval quality degradation, and a feedback loop where support agents mark incorrect answers for retraining. Plan for ongoing operations from day one.
- No fallback instruction. The LLM must be instructed to say "I could not find a relevant answer in the documentation" when the retrieved chunks are insufficient — not to guess. This is the primary guard against hallucination and must be explicit in the system prompt.
iZenTech's approach to RAG projects
We approach RAG projects in four phases:
- Document audit and ingestion design. Before writing any code, we review the source documents — their formats, update frequency, access control requirements, and quality. The ingestion pipeline design is informed by this audit, not a generic template.
- Baseline evaluation. We build a golden question set from real user queries (or approximate ones from stakeholders), run the first RAG version against it, and establish the faithfulness and relevance baseline. No system goes live without meeting a defined accuracy threshold.
- Iterative retrieval improvement. Chunk size, embedding model, re-ranking, hybrid search — these are tuned iteratively against the evaluation set, not guessed. Most production RAG systems reach their target accuracy in 2–4 tuning iterations.
- Operations handoff. We deliver monitoring dashboards (query volume, retrieval latency, confidence scores), document refresh pipelines, and a runbook for the client team to maintain the knowledge base. A RAG system is not a one-time deliverable — it is a living product.
Frequently asked questions
RAG is a technique that improves large language model (LLM) responses by first retrieving relevant information from a private knowledge base, then feeding that information to the LLM as context before generating an answer. Instead of the model relying solely on patterns learned during training, it reads actual source documents at query time. This gives the model access to up-to-date, domain-specific, or confidential information it was never trained on, dramatically reducing hallucinations and improving accuracy.
Fine-tuning teaches the model new behaviour or style by retraining it on additional data — expensive, time-consuming, and requires re-training when the data changes. RAG keeps the base model frozen and retrieves fresh information at query time from a vector database. RAG is significantly cheaper, updates instantly when source documents change, and can cite its sources. For most business knowledge-base and Q&A use cases, RAG is the right choice. Fine-tuning is better when you want to change how the model writes or reasons, not just what it knows.
A vector database stores documents as numerical embeddings — high-dimensional vectors that capture semantic meaning. When a user asks a question, RAG converts the question into an embedding and searches the vector database for the most semantically similar document chunks, not just keyword matches. This means RAG finds relevant passages even when the user's words differ from the source document's wording. Popular vector databases include Pinecone, Weaviate, Chroma, and pgvector (PostgreSQL extension).
A focused RAG application — for example, a customer support bot trained on your product documentation — typically costs ₹1,50,000 to ₹4,00,000 to build, including document ingestion pipeline, vector store setup, retrieval logic, and a chat interface. Monthly running costs are ₹10,000–₹31,000 at SME query volumes (around 5,000 queries/month). Enterprise RAG systems with multiple document sources, access control, and analytics dashboards start at ₹5,00,000–₹12,00,000.
Yes. Modern multilingual embedding models (such as multilingual-e5-large and OpenAI's text-embedding-3-large) produce high-quality embeddings for Hindi, Tamil, Telugu, Bengali, Gujarati, Marathi, and other Indian languages. The LLM generation step works well in Hindi with models like GPT-4o and Claude. A RAG system built today can handle Indian-language documents, respond in the user's language, and mix languages within a single query — making it practical for companies with multilingual customer bases or documents.