Scrape Reddit Posts Using PMAW & Python

Python

In this snippet the PMAW library is used to scrape 60,000 posts from the technology subreddit between 1st October 2019 and 1st October 2021. The results are used to create a dataframe which is then output to a csv file.

 1|  import pandas as pd
 2|  from pmaw import PushshiftAPI
 3|  import datetime as dt
 4|  import os.path as path
 5|  
 6|  api = PushshiftAPI()
 7|  
 8|  before = int(dt.datetime(2021,10,1,0,0).timestamp())
 9|  after = int(dt.datetime(2019,10,1,0,0).timestamp())
10|  
11|  subreddit="technology"
12|  limit=60000
13|  posts = api.search_submissions(subreddit=subreddit, limit=limit, before=before, after=after)
14|  posts_df = pd.DataFrame(posts)
15|  filepath = path.abspath(path.join(__file__ ,'../..','data/raw/technology_posts.csv'))
16|  posts_df.to_csv(filepath,index=False)
Greg He
Scraping | Reddit | Pmaw
Did you find this snippet useful?

Sign up for free to to add this to your code library

Remove Stop Words from Text in DataFrame Column
Python

Pandas | Nltk | Nlp

133
LightGBM Custom Loss Function
Python

LightGBM | Loss function

121
Tuning XGBoost Hyperparameters with Grid Search
Python

XGBoost | Tuning

116
How to Convert DataFrame Values Into Percentages
Python

Lambda | Percentages | Pipe

109
How to Scale Data Using Standard Scaler But Keep Column Names
Python

Scaler | Standard

78
LightGBM Hyperparameter Tuning with GridSearch
Python

Python | LightGBM | Hyperparameter tuning | Gridsearch

77
Calculating Root Mean Squared Error (RMSE) with Sklearn and Python
Python

Rmse

77
Dynamically Create Columns in Pandas Dataframe
Python

Pandas | Columns

61
How to Train a Catboost Classifier with GridSearch Hyperparameter Tuning
Python

Catboost | Hyperparameter tuning

60
Overwriting a Python Logging File Instead of Appending
Python

Logging

55