Sentiment Analysis Project in Python: From TF-IDF Baseline to Transformers

How to build a sentiment analysis project in Python, step by step

1. Step 1. Set up the environment

   Create a project folder and virtual environment, then install the libraries.

   mkdir sentiment-project && cd sentiment-project
   python -m venv .venv
   # Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
   pip install datasets scikit-learn transformers torch joblib streamlit

   On Google Colab, torch, transformers and scikit-learn are usually preinstalled; pip install anything that is missing, then continue from Step 2 in a notebook.

2. Step 2. Load the IMDB dataset from Hugging Face

   The Hugging Face datasets library downloads and caches the data on first use.

   # data.py
   import re
   from datasets import load_dataset
   
   def clean(text):
       text = re.sub(r"<br\s*/?>", " ", text)  # IMDB reviews contain <br /> tags
       return re.sub(r"\s+", " ", text).strip()
   
   def load_imdb():
       imdb = load_dataset("stanfordnlp/imdb")
       train, test = imdb["train"], imdb["test"]
       X_train = [clean(t) for t in train["text"]]
       X_test = [clean(t) for t in test["text"]]
       return X_train, train["label"], X_test, test["label"]
   
   if __name__ == "__main__":
       X_train, y_train, X_test, y_test = load_imdb()
       print(len(X_train), "train /", len(X_test), "test")
       print(X_train[0][:300], "| label:", y_train[0])  # 0 = negative, 1 = positive

   Always print a few raw examples before training. You will spot problems like HTML tags, very long reviews or label mix-ups in seconds.

3. Step 3. Train the TF-IDF and Logistic Regression baseline

   The baseline is your reference point. Every fancier model has to beat it to be worth its cost.

   # baseline.py
   import joblib
   from sklearn.pipeline import make_pipeline
   from sklearn.feature_extraction.text import TfidfVectorizer
   from sklearn.linear_model import LogisticRegression
   from sklearn.metrics import accuracy_score, f1_score, classification_report
   from data import load_imdb
   
   X_train, y_train, X_test, y_test = load_imdb()
   
   model = make_pipeline(
       TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=100_000, sublinear_tf=True),
       LogisticRegression(max_iter=1000),
   )
   model.fit(X_train, y_train)
   pred = model.predict(X_test)
   
   print("Accuracy:", round(accuracy_score(y_test, pred), 4))
   print("F1:", round(f1_score(y_test, pred), 4))
   print(classification_report(y_test, pred, target_names=["negative", "positive"]))
   joblib.dump(model, "baseline.joblib")

   Using word pairs (ngram_range=(1, 2)) lets the model learn phrases like "not good" and "waste of", which single words miss. Write down both numbers; they go in your comparison table.

4. Step 4. Look inside the baseline

   A big advantage of Logistic Regression is that you can see what it learned. Each word or phrase gets a weight: large positive weights push towards positive, large negative weights towards negative.

   # inspect_baseline.py
   import joblib
   
   model = joblib.load("baseline.joblib")
   tfidf, clf = model.named_steps["tfidfvectorizer"], model.named_steps["logisticregression"]
   words = tfidf.get_feature_names_out()
   weights = clf.coef_[0]
   
   order = weights.argsort()
   print("Most negative:", [words[i] for i in order[:15]])
   print("Most positive:", [words[i] for i in order[-15:][::-1]])

   Put these two lists in your report. They make the model explainable in a way a transformer is not, and examiners almost always ask "how does your model decide?"

5. Step 5. Run a pretrained transformer with the pipeline API

   The transformers pipeline loads a model already fine-tuned for sentiment, so there is no training step. We run it on a 1,000-review sample, because running all 25,000 on a CPU takes a long time.

   # transformer_eval.py
   import random
   from transformers import pipeline
   from sklearn.metrics import accuracy_score, f1_score
   from data import load_imdb
   
   _, _, X_test, y_test = load_imdb()
   random.seed(42)
   idx = random.sample(range(len(X_test)), 1000)
   texts = [X_test[i] for i in idx]
   labels = [y_test[i] for i in idx]
   
   clf = pipeline("sentiment-analysis",
                  model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
   out = clf(texts, truncation=True, batch_size=16)
   pred = [1 if o["label"] == "POSITIVE" else 0 for o in out]
   
   print("Transformer accuracy:", round(accuracy_score(labels, pred), 4))
   print("Transformer F1:", round(f1_score(labels, pred), 4))

   truncation=True cuts reviews longer than the model's 512-token limit instead of crashing. For a fair comparison, also score the baseline on the same 1,000 reviews with joblib.load("baseline.joblib").predict(texts).

6. Step 6. Compare the models and study the mistakes

   Build one table for your report with both models on the same 1,000 reviews:

   Model	Accuracy	F1	Time for 1,000 reviews	Size
   TF-IDF + Logistic Regression	your number	your number	your number	a few MB
   DistilBERT (SST-2)	your number	your number	your number	about 270 MB

   Then read 20 reviews each model got wrong. You will usually find patterns: sarcasm, reviews that praise the actors but hate the plot, negation far from the opinion word, or very long reviews where truncation cut the verdict. Writing two paragraphs about these error patterns shows more understanding than a one-point accuracy gain.

   Note that the DistilBERT model was fine-tuned on short movie review sentences (SST-2), not on IMDB itself. That mismatch is a good discussion point, and fine-tuning it on IMDB training data is a natural upgrade. See what fine-tuning is before you try it.

7. Step 7. Build a Streamlit demo

   A live demo where the examiner types their own review is far more convincing than screenshots.

   # app.py
   import joblib
   import streamlit as st
   from transformers import pipeline
   
   @st.cache_resource
   def load_models():
       baseline = joblib.load("baseline.joblib")
       bert = pipeline("sentiment-analysis",
                       model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
       return baseline, bert
   
   baseline, bert = load_models()
   st.title("Sentiment Analyser")
   text = st.text_area("Type a review", "The phone looks great but the battery dies by lunch.")
   
   if st.button("Analyse") and text.strip():
       p = baseline.predict_proba([text])[0][1]
       b = bert(text, truncation=True)[0]
       col1, col2 = st.columns(2)
       col1.metric("TF-IDF + LR", "Positive" if p >= 0.5 else "Negative", f"{p:.2f} positive")
       col2.metric("DistilBERT", b["label"].title(), f"{b['score']:.2f} confidence")

   Run it with streamlit run app.py. Mixed reviews like the default example are where the two models disagree, which makes for a good live demonstration.

Frequently asked questions

Keep learning

References

• IMDB dataset on Hugging Face (stanfordnlp/imdb)
• Maas et al., Large Movie Review Dataset (Stanford AI Lab)
• Hugging Face Transformers: pipelines documentation
• DistilBERT SST-2 sentiment model card
• Patwa et al., SemEval-2020 Task 9: Sentiment Analysis of Code-Mixed Tweets (arXiv)