Datasnips was built out of frustration in wasting time searching for Data Science & AI code snippets through old projects, documentation and internet sources.
We wanted a platform to easily find & organise our snippets, to share them with the community and discover those shared by other users.
At Datasnips you can quickly search, discover and bookmark snippets from the community, curate your own snippet library for easy reference and share your own snippets for others to use.
Our aim is for Datasnips to be the place where everyone can find the code snippet they need. Sign up to be part of the community.
Data Minification
A function that takes a dataframe and converts the data type of each numerical column to the optimum type based on the size of the values. This reduces the size in memory of the dataframe.
def minification(df):
for col in df.columns:
if is_numeric_dtype(df[col]):
if is_integer_dtype(df[col]):
max_value = df[col].max()
min_value = df[col].min()
if min_value >= 0:
if max_value < 255:
df[col] = df[col].astype(np.uint8)
elif max_value < 65535:
df[col] = df[col].astype(np.uint16)
elif max_value < 4294967295:
df[col] = df[col].astype(np.uint32)
else:
df[col] = df[col].astype(np.uint64)
else:
if min_value > np.iinfo(np.int8).min and max_value < np.iinfo(np.int8).max:
df[col] = df[col].astype(np.int8)
elif min_value > np.iinfo(np.int16).min and max_value < np.iinfo(np.int16).max:
df[col] = df[col].astype(np.int16)
elif min_value > np.iinfo(np.int32).min and max_value < np.iinfo(np.int32).max:
df[col] = df[col].astype(np.int32)
elif min_value > np.iinfo(np.int64).min and max_value < np.iinfo(np.int64).max:
df[col] = df[col].astype(np.int64)
else:
df[col] = df[col].astype(np.float32)
return df
minification(df)By detro - Last Updated Nov. 23, 2020, 8:26 p.m.
Very useful, thanks
Nlp - String - Punctuation
DatasnipsNov. 23, 2020, 10:10 p.m.