Remove Stop Words from Text in DataFrame Column

Python

This code snippet gives an example of how to remove stop words such as "the", "at" etc from columns in a Pandas dataframe that contains text. This is an important early cleaning step before transforming text data into a bag of words for NLP modelling.

Here we have a dataframe with a column named "tweet" that contains tweet text data. We use the Pandas apply with the lambda function along with list comprehension to remove stop words as declared in the NLTK library.

 1|  import nltk
 2|  nltk.download('stopwords')
 3|  from nltk.corpus import stopwords
 4|  
 5|  stop_words = stopwords.words('english')
 6|  df['tweet'] = df['tweet'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop_words)]))
detro
Pandas | Nltk | Nlp
Did you find this snippet useful?

Sign up for free to to add this to your code library

LightGBM Custom Loss Function
Python

LightGBM | Loss function

127
Tuning XGBoost Hyperparameters with Grid Search
Python

XGBoost | Tuning

122
How to Convert DataFrame Values Into Percentages
Python

Lambda | Percentages | Pipe

115
How to Scale Data Using Standard Scaler But Keep Column Names
Python

Scaler | Standard

86
LightGBM Hyperparameter Tuning with GridSearch
Python

Python | LightGBM | Hyperparameter tuning | Gridsearch

85
Calculating Root Mean Squared Error (RMSE) with Sklearn and Python
Python

Rmse

82
Dynamically Create Columns in Pandas Dataframe
Python

Pandas | Columns

65
How to Train a Catboost Classifier with GridSearch Hyperparameter Tuning
Python

Catboost | Hyperparameter tuning

65
Overwriting a Python Logging File Instead of Appending
Python

Logging

61
Reading CSV Files from Amazon S3 into Pandas Dataframes
Python

Pandas | Python | S3

56