RAG Interview Questions and Answers for Freshers and Experienced Engineers

A RAG pipeline has two phases. In indexing, you load documents, split them into chunks, convert each chunk into an embedding and store it in a vector index with metadata. At query time, you embed the user's question, retrieve the most similar chunks, insert them into the prompt with instructions, and let the LLM generate an answer grounded in that context, ideally with citations. Most real systems add steps such as query rewriting and reranking between retrieval and generation.

Because most document sets are far larger than any context window, and even when they fit, sending everything on every request is slow and expensive. Models also use very long contexts unevenly and can miss details buried in the middle. RAG sends only the few passages that matter, which keeps cost and latency low and makes answers easier to cite. For a small, fixed document set, stuffing the full text into the context window can be a sensible shortcut.

Chunking splits documents into smaller passages that can be embedded and retrieved individually. Chunk size is a trade-off: small chunks are precise but may lose context, while large chunks keep context but dilute the embedding and waste prompt space. Many teams start around a few hundred tokens with some overlap, then tune using a test set. The right size depends on the content: FAQ answers suit small chunks, legal clauses and technical manuals often need larger ones.

Common strategies are fixed-size chunks with overlap, recursive splitting that respects paragraphs and sentences, structure-based splitting by headings or sections, and semantic chunking that splits where the topic changes. A useful advanced pattern is parent-child (small-to-big) retrieval: search over small chunks for precision, then send the larger parent section to the model for context. For PDFs with tables, keep each table intact as its own chunk rather than cutting it in half.

Choose by measured retrieval quality on your own data, not by popularity. Check language support (important if users write in Hindi or Hinglish), the maximum input length, vector dimension (which affects storage and speed), cost or hosting needs, and whether the model handles your domain vocabulary. Public leaderboards give a shortlist, but run a quick recall test on 30 to 50 of your own questions. Always use the same model for documents and queries. See embeddings.

The common ones are cosine similarity, dot product and Euclidean distance. Cosine similarity compares the direction of two vectors and ignores length, which suits most text embeddings. When vectors are normalised to unit length, cosine, dot product and Euclidean distance all produce the same ranking, so the choice mostly matters for unnormalised vectors. Use the metric the embedding model was trained with, which its documentation usually states.

Top-k is the number of chunks the retriever returns for each query. A small k keeps the prompt short and focused but risks missing the answer; a large k improves recall but adds noise, cost and latency. A common pattern is to retrieve a larger set, say 20 to 50, then rerank and send only the best 3 to 8 to the model. Pick the final k by measuring answer accuracy on a test set at a few values.

Approximate nearest neighbour (ANN) search finds vectors that are very likely the closest to a query without comparing against every vector, trading a little accuracy for large speed gains. HNSW (Hierarchical Navigable Small World) builds a layered graph where upper layers have few long-range links and lower layers have dense local links; search starts at the top and greedily moves closer layer by layer. Other approaches include IVF, which clusters vectors and searches only the nearest clusters. More in vector databases.

Hybrid search runs keyword search (usually BM25) and vector search together and merges the results. Vector search understands meaning and paraphrases, while keyword search is better at exact terms like product codes, error IDs, names and rare jargon. A common way to merge is reciprocal rank fusion, which scores each document by its rank in each list, so you do not need to normalise very different score scales. Hybrid search is often the cheapest big improvement to a weak RAG system.

Reranking takes the top results from the first retrieval stage and reorders them with a more accurate model. Embedding retrievers are bi-encoders: they encode query and document separately, which is fast but loses fine detail. A cross-encoder reads the query and document together and scores their relevance directly, which is more accurate but too slow to run over the whole corpus. So you retrieve broadly and cheaply, then rerank a short list. The cost is added latency per query.

Query rewriting uses an LLM to turn the user's raw question into better search queries before retrieval. Examples include fixing vague or misspelt queries, generating several variants (multi-query), splitting a complex question into sub-questions, and HyDE, where the model writes a hypothetical answer and you embed that instead. In chatbots, it also turns follow-ups like "what about the second one?" into standalone queries. The trade-off is an extra model call, which adds cost and latency.

Metadata filtering restricts search to chunks that match structured attributes such as document type, date, product, region, language or user permission. It removes whole categories of wrong answers, for example retrieving a 2022 leave policy when only the current one is valid. Filters can apply before the vector search or after it; pre-filtering is usually more accurate but can be slower in some vector stores. Plan your metadata during ingestion, because adding it later means re-indexing.

Parsing quality decides retrieval quality, so treat it as a first-class step. Use a layout-aware parser for PDFs so headings, columns and tables are preserved, and run OCR on scanned pages. Convert tables to a clean text or markdown form and keep each table in one chunk with its caption. Spot-check parsed output manually, because broken text from multi-column layouts is a very common hidden cause of bad answers in Indian government and banking PDFs.

Give each retrieved chunk an ID and source metadata (document name, page, section), include those IDs in the prompt, and instruct the model to cite the IDs it used for each claim. Then verify in code that every cited ID was actually retrieved and drop or flag answers with missing or invalid citations. Citations build user trust and make debugging much faster. The limitation is that a citation shows where text came from, not that the model used it faithfully.

Lost in the middle refers to research showing that LLMs use information at the start and end of a long context more reliably than information in the middle. In RAG, this means stuffing 20 chunks into the prompt can hide the one that matters. Mitigations: send fewer, better chunks after reranking, place the most relevant chunks first or last, and remove near-duplicates. It is another reason more context is not automatically better.

Evaluate retrieval and generation separately. For retrieval, build a test set of questions with known relevant passages and measure recall at k, precision at k and ranking metrics such as MRR or nDCG. For generation, measure faithfulness (is every claim supported by the retrieved context?), answer relevance (does it address the question?) and correctness against reference answers. Open-source frameworks such as Ragas automate some of these with LLM judges. Validate the automated scores against human review on a sample.

Faithfulness checks whether the answer is supported by the retrieved context, with no invented claims. Answer relevance checks whether the answer actually addresses the user's question. They fail independently: an answer can be perfectly faithful to the context but answer a different question, or relevant and fluent but partly made up. Tracking both tells you whether to fix grounding or fix retrieval and prompting. Low faithfulness is a hallucination signal.

Agentic RAG lets an LLM agent decide how to retrieve instead of following one fixed search step. The agent can choose which source or tool to query, rewrite the query, retrieve again if results look weak, and combine answers from several searches for multi-step questions. It handles complex questions better than single-shot RAG, but adds latency, cost and new failure modes such as loops. Use it when questions genuinely need multiple hops. See our agentic AI interview questions.

Graph-based RAG builds a knowledge graph of entities and relationships from your documents, often with an LLM doing the extraction, and retrieves over that graph instead of, or alongside, text chunks. It helps with questions that connect facts across many documents, such as "which suppliers are linked to delayed projects?", and with broad questions about themes in a whole collection. The trade-off is a much more expensive and complex indexing step, so start with standard RAG and add a graph only when you see these question types failing.

Give every document a stable ID and a content hash, and run incremental ingestion: re-chunk and re-embed only documents whose hash changed, and delete chunks of removed documents. Store timestamps and version metadata so you can filter out superseded content. Schedule regular syncs from the source systems, and monitor for stale answers. If you change the embedding model, you must re-embed the entire corpus, because vectors from different models are not comparable.

Cut tokens first: send fewer, reranked chunks and trim boilerplate from them. Cache embeddings for repeated queries and cache final answers for frequent questions. Use a smaller, cheaper model for query rewriting and a larger one only for the final answer, or skip rewriting when the query is already clear. Tune the vector index for speed, stream the answer and run retrieval steps in parallel where possible. Measure cost per correct answer, not cost per query.

RAG is the wrong choice when the problem is behaviour rather than knowledge, such as a fixed output style or a classification task, where prompting or fine-tuning fits better. It is also a poor fit for questions that need aggregations over structured data, like "total sales by state last quarter", which should go to SQL instead. And for a small, stable document set, putting the whole text in the prompt may be simpler. Good candidates know when not to build the pipeline.

Treat it as a retrieval failure until proven otherwise. Take the failing questions and check whether the correct passage was even indexed: parsing errors and skipped files are common. Then check whether it was retrieved in the top 50; if yes but not in the final top 5, add or tune a reranker. If not retrieved at all, look at chunking, try hybrid search and query rewriting, and test a different embedding model. Add every failing question to your test set.

Exact identifiers are a known weakness of pure vector search, because embeddings capture meaning, not exact strings. Add keyword search with BM25 and combine it with vector results through hybrid search. Also extract IDs into metadata fields during ingestion, so you can filter directly when the query contains a recognisable code pattern. Measure the fix with a set of ID-based test queries.

Clarify users, languages and permissions first. Ingest with a layout-aware parser plus OCR, chunk by policy section, and tag metadata: policy type, effective date, location, language and access level. Use a multilingual embedding model, hybrid search and a reranker. Answer in the user's language with citations, and say "I could not find this" when retrieval is weak. Enforce document permissions at retrieval time, and evaluate with a bilingual test set reviewed by HR.

The most likely cause is that the index still contains vectors from the old model while queries use the new one, or the index was only partly re-embedded. Vectors from different models live in different spaces and cannot be compared. Re-embed the full corpus, then rerun your retrieval test set to confirm the new model is actually better on your data. Next time, build the new index in parallel and switch only after it passes evaluation.

Instruct the model to answer only from the provided context and to reply with a clear fallback when the context does not contain the answer. Add a relevance threshold so that if the best retrieved chunks score too low, you skip generation and return the fallback. A faithfulness check after generation can catch unsupported claims. Test with a set of deliberately out-of-scope questions and track how often the bot correctly refuses.

Access control was missing at retrieval time. The bot searched every document regardless of who was asking. Fix it by storing permission metadata, such as allowed groups, with every chunk and filtering on the user's permissions before or during retrieval, never after generation. Also treat retrieved documents as untrusted input, since hidden instructions inside them can cause prompt injection. Audit logs and a security review before launch are standard practice.