Python Pandas - Pickling

---

---

Pickling in Python, also known as serialization, is the process of converting a Python object into a byte stream, which can be saved and later deserialized back into the original object structure. In the context of Pandas, pickling enables the efficient saving and loading of DataFrames and Series objects.

The Python Pandas library provides easy to use functions for pickling DataFrames and Series objects using its to_pickle() and read_pickle() methods. These methods use Python's cPickle module, which implements a binary format for efficiently saving data structures to disk and loading them back to the Pandas object using the pickle format.

In this tutorial, we will learn about how to use Pandas built-in pickling functionalities for efficient serialization and deserialization of Pandas objects with more customization options.

Saving a Pandas Object to a Pickle File

To pickle the Pandas objects such as DataFrame, or Series, you can use the to_pickle() method, which saves them to a file in pickle format.

Example: Saving a DataFrame to a Pickle File

Following is the example that uses the to_pickle() method for saving a Pandas DataFrame object into a pickle file.

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({
"A": [1, 2, 3],
"B": [4, 5, 6]
})

# Display the Input DataFrame
print("Original DataFrame:")
print(df)

# Save the DataFrame to a pickle file
df.to_pickle("dataframe.pkl")

print('\nDataFrame is successfully saved as a pickle file named "dataframe.pkl".')

When we run above program, it produces following result −

Original DataFrame:

	A	B
0	0	4
1	1	5
2	2	6

DataFrame is successfully saved as a pickle file named 'dataframe.pkl'.

If you visit the folder where the pickle files are saved, you can observe the generated pickle file.

Example: Saving a Series to a Pickle File

Following is the example that uses the to_pickle() method for saving a Pandas Series object into a pickle file.

import pandas as pd

# Creating a Pandas Series
s = pd.Series([1, 2, 3, 4])

# Display the Input Series
print("Original Series:")
print(s)

# Save the Series as a pickle file
s.to_pickle("series_to_pickle_file.pkl")

print('\nSeries is successfully saved as a pickle file named "series_to_pickle_file.pkl".')

When we run above program, it produces following result −

Original Series:
0      1
1      2
2      3
3      4
dtype: int64

Series is successfully saved as a pickle file named "series_to_pickle_file.pkl".

In this example, the Series object is saved to a file named "series_to_pickle_file.pkl" in the current directory.

Loading a Pickled Pandas Object

For loading a pickled file into the Pandas object (Series, or DataFrame objects), you can use the read_pickle() method from Pandas. This will deserialize the byte stream and recreate the Pandas object.

It's important to note that, loading pickled data from untrusted sources can be risky. Always verify the source before deserializing a pickle file.

Example

This example loads a Pandas Series object from a pickle file using the Pandas read_pickle() method.

import pandas as pd

# Creating a Pandas Series
s = pd.Series([1, 2, 3, 4], index=["cat", "dog", "fish", "mouse"])

# Display the Input Series
print("Original Series:")
print(s)

# Save the Series as a pickle file
s.to_pickle("series_read_pickle_file.pkl")

# Load the Series from the pickle file
unpickled_series = pd.read_pickle("series_read_pickle_file.pkl")
print("\nLoaded Series:")
print(unpickled_series)

While executing the above code we get the following output −

Original Series:
cat      1
dog      2
fish     3
mouse    4
dtype: int64

Loaded Series:
cat      1
dog      2
fish     3
mouse    4
dtype: int64

Working with Compressed Pickle Files

Pandas pickling functionality also supports reading and writing compressed pickle files. The following compression formats are supported:

• gzip

• bz2

• xz

• zstd

You can specify the compression format either explicitly or by inferring it from the file extension. To do this, set the compression parameter of the to_pickle() and read_pickle() methods to 'infer'.

Example: Saving a Compressed Pickle File

Here is an example of demonstrating how to save a DataFrame to a compressed pickle file using gzip compression.

import pandas as pd

# Create a DataFrame
df = pd.DataFrame({"Col_1": range(5), "Col_2": range(5, 10)})
print("Original DataFrame:")
print(df)

# Save the DataFrame to a pickle file with gzip compression
df.to_pickle("dataframe_compressed.pkl.gz", compression="gzip")

print("\nDataFrame is successfully saved as a pickle file with gzip compression.")

Following is an output of the above code −

Original DataFrame:

	Col_1	Col_2
0	0	5
1	1	6
2	2	7
3	3	8
4	4	9

DataFrame is successfully saved as a pickle file with gzip compression.

Example: Loading a Compressed Pickle File

The following example demonstrates how to use the read_pickle() method to read the gzip compressed pickle file into a Pandas object. To load a compressed pickle file, you simply need to specify the compression type.

import pandas as pd

# Create a DataFrame
df = pd.DataFrame({"Col_1": range(5), "Col_2": range(5, 10)})
print("Original DataFrame:")
print(df)

# Save the DataFrame to a pickle file with gzip compression
df.to_pickle("dataframe_compressed.pkl.gz", compression="gzip")

# Load the compressed file
compressed_df = pd.read_pickle("dataframe_compressed.pkl.gz")
print("\nLoaded Compressed DataFrame:")
print(compressed_df)

Following is an output of the above code −

Original DataFrame:

	Col_1	Col_2
0	0	5
1	1	6
2	2	7
3	3	8
4	4	9

Loaded Compressed DataFrame:

	Col_1	Col_2
0	0	5
1	1	6
2	2	7
3	3	8
4	4	9

Inferring Compression Type from a Pickle File Extension

PaPandas will automatically detect the compression type if the compression parameter is set to infer and the file ends with .gz, .bz2, .xz, or .zst extensions.

Example

This example sets the compression parameter value to infer for automatically inferring the compression type from the file extension.

import pandas as pd

# Create a DataFrame
df = pd.DataFrame({"Col_1": [1, 2, 3, 4, 5],
"Col_2": ["a", "b", "c", "d", "e"]})

print("Original DataFrame:")
print(df)

# Save with inferred compression type
df.to_pickle("dataframe.pkl.xz", compression="infer")

# Load with inferred compression type
df_loaded_inferred = pd.read_pickle("dataframe.pkl.xz", compression="infer")

print("\nLoaded with inferred compression type:")
print(df_loaded_inferred)

Following is an output of the above code −

Original DataFrame:

	Col_1	Col_2
0	1	a
1	2	b
2	3	c
3	4	d
4	5	e

Loaded with inferred compression type:

	Col_1	Col_2
0	1	a
1	2	b
2	3	c
3	4	d
4	5	e