Sentiment Analysis in 3 steps: Using quick_sentiments in Python
This blog is based on the Python package quick_sentiments that I developed to streamline sentiment analysis. You can view the code here.
Major Updates: Version 3.5 and above handles pre_processing() differently. Please refer to the 2nd example if you are using the latest version.
What if you could run a full, end-to-end sentiment analysis pipeline — from raw text to a final prediction — in just three simple steps? That’s the exact problem I designed my new Python package, quick_sentiments, to solve. It handles all the complex and time-consuming tasks of text cleaning, vectorization, and machine learning ‘under the hood’, letting you focus on your results, not the boilerplate.
If you’ve built a sentiment analysis project from scratch, you know the initial hurdle. You’re juggling imports from nltk, scikit-learn, and others, and hoping that a library update doesn’t break your environment. quick_sentiments is designed to solve this. It abstracts away the setup, managing all the dependencies and complex function calls for you.
You can install the package in Python using the code below
!pip install quick-sentiments
#OR
git clone https://github.com/AlabhyaMe/quick_sentiments.gitpSTEP 1: Preprocessing
So, let’s dive into the first step: preprocessing. Before a model can understand text, it must be cleaned. This is a critical but tedious stage that involves a checklist of tasks: removing HTML tags, stripping out emojis and punctuation, handling stopwords, lemmatization, and standardizing case. A clean text is the necessary foundation for any high-quality vectorization and, ultimately, an accurate model.
As you can see from the example, using pre_process_spacy or pre_process_nltkfunction is incredibly simple.
Updates: From version 0.4.1 onwards, users have the option to choose between ntlk or spacy pre-processing. Spacy is faster for large datasets.
# version 3.4 or less
import polars as pl
#IMPORT pre_process from quick_sentiments
from quick_sentiments import pre_process
df_train = pl.read_csv("yourpath.csv",encoding='ISO-8859-1')
response_column = "reviewText" #based on the column name
sentiment_column = "sentiment"
#USING pre_process
df_train = df_train.with_columns(
pl.col(response_column).map_elements(lambda x: pre_process(x, remove_brackets=True)).alias("processed") #add inside the map_elements
)# version 3.5 onwards
from quick_sentiments import pre_process
df_train = pl.read_csv("yourpath.csv",encoding='ISO-8859-1')
response_column = "reviewText" #based on the column name
# lamda is handled inside the function now. the function returns
# dataset with new column
# with version 0.4.1 onwards, user have options to either use nltk or spacy for pre processing
df_train = pre_process_spacy(df_train, text_column=response_column, new_column_name="processed")
df_train = pre_process_nltk(df_train, text_column=response_column, new_column_name="processed")Think about what it usually takes to clean text from scratch. You’d be importing re for regex to handle URLs, another regex for HTML tags, importing nltk to download and initialize a stopword list, another nltk module for a lemmatizer... it's a long and tedious setup that you have to write for every single project.
quick_sentiments condenses that entire process into one highly configurable function. Just look at the control you have right out of the box:
def pre_process_spacy(doc,
remove_brackets=True,
remove_urls=True,
remove_html=True,
lemmatize=True,
remove_stop_words=True,
remove_nums=False,
... # and many more
return_string=True):
#THE CODE FOR ALL THESE IS WRITTEN FOR YOU INSIDE THE PACKAGE. You can also view the code here.
This single function is a complete cleaning pipeline. In our polars example, we used the defaults but specifically set remove_brackets=True. What if your specific task needs to keep numbers in the text? Just pass remove_nums=False. Want to skip the lemmatization step to speed things up? lemmatize=False.
From unicode normalization and emoji removal to tokenizing and lemmatization, pre_process handles the entire checklist in one clean, simple call.
This clean, tokenized string is now the perfect foundation for our next step: vectorization.
STEP 2: Data splitting, Running vectorization, ML Models, Accuracy
Now we get to the powerhouse of the quick_sentiments package. After preprocessing, you’re left with the “messy middle” of machine learning: data splitting, vectorization, model training, and evaluation. This is where most projects slow down, and it’s where run_pipeline() takes over.
This single function handles all that complexity for you. Crucially, it performs the train/test split before vectorization, which is the correct way to prevent data leakage and get a reliable measure of your model’s performance.
All you have to do is provide your data, name your columns, and — most importantly — choose your experiment.
Want to test a classic TF-IDF with Logistic Regression? Or jump straight to 100-dimensional GloVe embeddings with a Neural Network? Or push it further with 300-dimensional Word2Vec? You just have to ask.
Here is a full list of the vectorization methods and ML models that are pre-built and ready for you to call by name:
- Vectorization Methods (
vectorizer_name): "bow": Bag-of-Words"tf": Term Frequency"tfidf": TF-IDF"wv": Word2Vec (trained on your data) (Will download if you don’t already have it)"glove_25","glove_50","glove_100","glove_200": Pre-trained GloVe embeddings in various dimensions- Machine Learning Models (
model_name): "logit": Logistic Regression"rf": Random Forest"xgb": XGBoost"nn": A simple Neural Network
All of this power is wrapped into one simple function call:
dt= run_pipeline(
vectorizer_name="glove_25", # BOW, tf, tfidf, wv, glove_25,glove_50, glove_100, gl0ve_200
model_name="logit", # logit, rf, XGB, nn
df=df_train,
text_column_name="processed", # this is the column name of the text data,
sentiment_column_name = "sentiment",
perform_tuning = False # make this true if you want to perform hyperparameter tuning, it will take longer time and
# may run out of memory if the dataset is large,
)Find the code here.
#THIS IS WHAT run_pipeline returns. It is important for prediction
return {
"model_object": trained_model_object,
"vectorizer_name": vectorizer_name,
"vectorizer_object": fitted_vectorizer_object,
"label_encoder": label_encoder,
"y_test": y_test,
"y_pred": y_pred,
"accuracy": accuracy_score(y_test, y_pred),
"report": classification_report(y_test, y_pred, output_dict=True, target_names=label_encoder.classes_)
}Another key feature of run_pipeline isn’t just what it does, but what it gives back to you. When the function is finished, it returns a single, powerful Python dictionary containing all your trained “artifacts.”
This means the fitted vectorizer, the trained model, the label encoder, and even the column names are all packaged up and returned. This is incredibly important because it’s the “brain” you’ve just trained, saved in a box, ready to be used on new, unseen data.
So, at this point, our core sentiment analysis workflow is complete with just two functions. With just pre_process and run_pipeline, we’ve successfully cleaned, vectorized, trained, and evaluated a model on our initial dataset.
But what about making predictions on completely new data? This is where quick_sentiments makes the next step easy. While you will, of course, need to pass your new raw text through the same pre_process function, you won’t have to re-train anything. The “artifact” object that run_pipeline returned has all the fitted components you need to instantly apply the correct vectorization and model to your new data, ensuring the entire process is consistent from start to finish.
STEP 3: The Payoff — Predicting on New, Unseen Data
#PRE_PROCESS THE NEW DATA
df_train = pre_process_spacy(new_data, text_column=response_column, new_column_name="processed")We’ve successfully trained and evaluated a model on our dataset, and our run_pipeline() function returned a Python dictionary (which we saved as dt) containing all our trained "artifacts."
Now, it’s time to use that trained “brain” on completely new, real-world data.
This is the final step, and quick_sentiments makes it just as simple as the first two. After you've passed your new, raw text through the same pre_process() function (to ensure the cleaning is identical), you're ready for the final, one-line command:
# The 'dt' object is the dictionary returned from our run_pipeline() call
predictions_df = make_predictions(
new_data=new_data,
text_column_name="processed",
vectorizer=dt["vectorizer_object"],
best_model=dt["model_object"],
label_encoder=dt["label_encoder"],
prediction_column_name="sentiment_predictions" # You can name the output column
)
#Version 0.4.3 makes this step simplier
predictions_df = make_predictions(
new_data=new_data,
text_column_name="processed",
prediction_column_name="sentiment_predictions", # You can name the output column
trained_results = dt # this is enough
# dt is the trained model from the pipeline
)What’s Happening “Under the Hood”
This one make_predictions() call is where the real power of the pipeline comes together. Let's look at the arguments to see why this is so reliable:
vectorizer=dt["vectorizer_object"]: You are passing in the actual fitted vectorizer that was trained on your data.best_model=dt["model_object"]: You are passing in the actual trained model.label_encoder=dt["label_encoder"]: You are passing in the fitted label encoder to correctly map the outputs back to your original labels (e.g., "positive", "negative").
Note, quick-sentiments 0.4.3onwards, you can just pass the dt as trained_results . It will handle vectorization, the best model, and label internally.
This design saves you from writing all the repetitive code for vectorizing and predicting. But more importantly, it eliminates the most common and dangerous errors in machine learning.
How? By passing in the fitted artifacts, you make it impossible to make a mistake. You don’t have to worry:
- “Did I re-fit my vectorizer on the new data by accident?”
- “Am I using the same
ngram_rangeandmax_featuresas my training data?" - “Am I using the model with the best hyperparameters, or just the default ones?”
The make_predictions() function guarantees that your new data is processed through the exact same, consistent pipeline as your original training data. The risk of error here is nullified because all the complex work is done for you "under the hood."
And with that, you have your new data frame with a new column of predictions. In just three major steps — pre_process(), run_pipeline(), and make_predictions()—we've completed the entire sentiment analysis journey from raw text to real-world, reliable predictions.
Find the full source code at: https://github.com/AlabhyaMe/quick_sentiments
