Apache Parquet

• Columnar: data stored by columns, not rows
• Compression: Snappy, Gzip, ZSTD, LZ4 per column
• Encoding: dictionary, RLE, bit-packing, delta for efficient storage
• Predicate pushdown: WHERE date > 2026-01-01 at reader level, not scan
• Schema embedded in file → self-describing

# Python (pandas)
import pandas as pd
df = pd.read_csv('sales.csv')
df.to_parquet('sales.parquet', compression='snappy')

# Read only needed columns (column pruning)
df = pd.read_parquet('sales.parquet', columns=['date', 'amount'])

# DuckDB — query parquet directly
duckdb.sql("SELECT SUM(amount) FROM 'sales.parquet' WHERE date > '2026-01-01'")

• Data Lake
• Iceberg
• Avro