Parquet

What it is

Apache Parquet is a binary, columnar file format for tabular data. It describes how data is laid out inside files; it is not a database, query engine, catalog, or transaction manager.

Why it matters

Analytical workloads commonly read a few columns across many rows. Keeping values for each column together reduces I/O, improves compression, and lets engines skip data that a query cannot use. Parquet is therefore a common storage format in data lakes on Amazon S3 and similar object stores.

How it works

CSV files store text rows and usually require parsing every field that is read. Row-based storage similarly keeps all fields for one record together, which is useful when an application frequently reads or changes complete records. Parquet instead groups values by column in column chunks and pages, with metadata describing their location and encoding.

For example, for SELECT AVG(salary) FROM employees, an engine can use column pruning to read the salary column without reading name, address, or other unrelated columns. Repeated values, similar values, and typed binary representations also make a column a strong target for compression and encoding.

Example

Spark, Amazon Athena, and Databricks can query Parquet stored in a data lake. The same files are useful as efficient training or feature data for analytics and machine-learning workflows because engines can scan only required columns and partitions.

Trade-offs and limitations

Parquet is optimized for large scans and batch writes, not frequent single-record updates. Updating one row generally means rewriting the file or files that contain it, then coordinating which file version readers should use. A table format such as Iceberg or Delta Lake adds that coordination layer; Parquet alone does not.

Lakehouse

Glossary

Amazon S3: Amazon Simple Storage Service, an object-storage service.
CSV: Comma-separated values, a plain-text tabular file format.
I/O: Input/output operations, such as reading from or writing to storage.