RAG Chatbot Project: Build a Chatbot on Your Own PDFs in Python

How to build a RAG chatbot in Python, step by step

1. Step 1. Set up the project

   Create a folder, a virtual environment and install four packages.

   mkdir rag-chatbot && cd rag-chatbot
   python -m venv .venv
   # Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
   pip install pypdf sentence-transformers numpy openai streamlit

   Make a docs folder and drop two or three PDFs into it. Start small: your syllabus, an exam rule book or a few research papers. Set your API key as an environment variable named OPENAI_API_KEY (or the variable your provider uses), never inside the code.

2. Step 2. Load the PDFs and split them into chunks

   Chunking decides what the retriever can find, so it matters more than most beginners expect. Chunks that are too big bury the answer; chunks that are too small lose context. Around 200 to 300 words with a little overlap is a good start.

   # rag.py
   from pathlib import Path
   from pypdf import PdfReader
   
   def load_chunks(folder="docs", size=250, overlap=50):
       chunks = []
       for pdf in Path(folder).glob("*.pdf"):
           reader = PdfReader(pdf)
           for page_no, page in enumerate(reader.pages, start=1):
               words = (page.extract_text() or "").split()
               step = size - overlap
               for start in range(0, max(len(words), 1), step):
                   text = " ".join(words[start:start + size])
                   if text.strip():
                       chunks.append({"text": text, "source": pdf.name, "page": page_no})
       return chunks

   Each chunk keeps its file name and page number. That small detail lets your chatbot show sources, which examiners and interviewers love.

3. Step 3. Create embeddings for every chunk

   An embedding model turns text into a vector so that similar meanings sit close together. all-MiniLM-L6-v2 is small, free and runs on a CPU.

   import numpy as np
   from sentence_transformers import SentenceTransformer
   
   embedder = SentenceTransformer("all-MiniLM-L6-v2")
   
   def build_index(chunks):
       texts = [c["text"] for c in chunks]
       vectors = embedder.encode(texts, normalize_embeddings=True, show_progress_bar=True)
       return np.array(vectors)

   Setting normalize_embeddings to True means a plain dot product now equals cosine similarity, which keeps the search code to one line. For a few hundred pages this step takes under a minute on a normal laptop.

4. Step 4. Retrieve the most relevant chunks

   Retrieval embeds the question and picks the chunks with the highest similarity score.

   def retrieve(question, chunks, vectors, k=4):
       q = embedder.encode([question], normalize_embeddings=True)[0]
       scores = vectors @ q
       best = np.argsort(scores)[::-1][:k]
       return [(chunks[i], float(scores[i])) for i in best]

   Print the retrieved chunks while you test. Most "the chatbot gave a wrong answer" bugs are really "the retriever fetched the wrong chunk" bugs, and you can only see that by looking at what was retrieved.

5. Step 5. Build the prompt and call the LLM

   The prompt does two jobs: it hands the model the retrieved context, and it tells the model to stay inside that context.

   import os
   from openai import OpenAI
   
   client = OpenAI()  # reads OPENAI_API_KEY from the environment
   MODEL = os.environ.get("LLM_MODEL", "gpt-4o-mini")  # use any current chat model from your provider's docs
   
   def answer(question, chunks, vectors):
       hits = retrieve(question, chunks, vectors)
       context = "\n\n".join(
           f"[{c['source']} p.{c['page']}] {c['text']}" for c, _ in hits
       )
       prompt = (
           "Answer the question using only the context below. "
           "If the answer is not in the context, say you could not find it. "
           "Mention the source in brackets.\n\n"
           f"Context:\n{context}\n\nQuestion: {question}"
       )
       reply = client.chat.completions.create(
           model=MODEL,
           messages=[{"role": "user", "content": prompt}],
           temperature=0,
       )
       return reply.choices[0].message.content, hits

   Temperature 0 keeps answers focused and repeatable, which is what you want when the job is to report facts from a document. Read what temperature does if you want to explain that choice.

6. Step 6. Add a chat interface with Streamlit

   Streamlit turns the functions above into a web app you can demo in a browser.

   # app.py
   import streamlit as st
   from rag import load_chunks, build_index, answer
   
   st.title("Ask my documents")
   
   @st.cache_resource
   def setup():
       chunks = load_chunks()
       return chunks, build_index(chunks)
   
   chunks, vectors = setup()
   
   if question := st.chat_input("Ask a question about your PDFs"):
       st.chat_message("user").write(question)
       text, hits = answer(question, chunks, vectors)
       with st.chat_message("assistant"):
           st.write(text)
           with st.expander("Sources"):
               for c, score in hits:
                   st.caption(f"{c['source']}, page {c['page']} (score {score:.2f})")

   Run it with streamlit run app.py. The cache_resource decorator means the PDFs are embedded once, not on every message.

7. Step 7. Test it and measure accuracy

   A project with numbers beats a project with screenshots. Write 20 questions whose answers you know are in the documents, plus 5 whose answers are not.

   For each question, record three things in a simple spreadsheet:

   1. Retrieval hit: did the right chunk appear in the top 4?
   2. Answer correct: was the final answer right?
   3. Refusal correct: for the 5 trick questions, did the bot say it could not find the answer?

   Your results line in the report then reads something like "retrieval hit rate 18 of 20, answer accuracy 17 of 20, correct refusals 5 of 5". Those three numbers are worth more in a viva than any amount of description, and they point straight at what to improve next.

Frequently asked questions

Keep learning

References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
• Sentence Transformers documentation
• pypdf documentation
• Streamlit documentation: build a basic LLM chat app