Scrape Reddit Posts Using PMAW & Python

Python

In this snippet the PMAW library is used to scrape 60,000 posts from the technology subreddit between 1st October 2019 and 1st October 2021. The results are used to create a dataframe which is then output to a csv file.

 1|  import pandas as pd
 2|  from pmaw import PushshiftAPI
 3|  import datetime as dt
 4|  import os.path as path
 5|  
 6|  api = PushshiftAPI()
 7|  
 8|  before = int(dt.datetime(2021,10,1,0,0).timestamp())
 9|  after = int(dt.datetime(2019,10,1,0,0).timestamp())
10|  
11|  subreddit="technology"
12|  limit=60000
13|  posts = api.search_submissions(subreddit=subreddit, limit=limit, before=before, after=after)
14|  posts_df = pd.DataFrame(posts)
15|  filepath = path.abspath(path.join(__file__ ,'../..','data/raw/technology_posts.csv'))
16|  posts_df.to_csv(filepath,index=False)
Greg He
Scraping | Reddit | Pmaw
Did you find this snippet useful?

Sign up for free to to add this to your code library

Remove Stop Words from Text in DataFrame Column
Python

Pandas | Nltk | Nlp

149
LightGBM Custom Loss Function
Python

LightGBM | Loss function

132
Tuning XGBoost Hyperparameters with Grid Search
Python

XGBoost | Tuning

127
How to Convert DataFrame Values Into Percentages
Python

Lambda | Percentages | Pipe

119
How to Scale Data Using Standard Scaler But Keep Column Names
Python

Scaler | Standard

92
LightGBM Hyperparameter Tuning with GridSearch
Python

Python | LightGBM | Hyperparameter tuning | Gridsearch

91
Calculating Root Mean Squared Error (RMSE) with Sklearn and Python
Python

Rmse

86
Dynamically Create Columns in Pandas Dataframe
Python

Pandas | Columns

68
How to Train a Catboost Classifier with GridSearch Hyperparameter Tuning
Python

Catboost | Hyperparameter tuning

68
Overwriting a Python Logging File Instead of Appending
Python

Logging

65