Skip to content

Parquet Update Strategy

Question

Why is Parquet a poor choice for frequent single-record updates, and what strategy would you use instead?

Short interview answer

Parquet is a columnar file format optimized for analytical scans and batch writes, not in-place row updates. Updating one record usually requires rewriting an affected file. Use a table format such as Iceberg or Delta Lake to coordinate rewritten files as atomic table versions, or choose an operational database when frequent point updates are the primary workload.

Detailed answer

An employees Parquet file stores values by column. For an analytical query, that is exactly what we want:

SELECT AVG(salary) FROM employees;

Query needs:       salary
Parquet reads:     salary column
Parquet skips:     employee_id, name, address, department

Reading one column instead of every field reduces I/O and often compresses better. This is why Parquet is a strong format for reports and large scans.

Now change one employee's address:

employees-part-08.parquet
  employee_id=41, address="Berlin"
  employee_id=42, address="Gdansk"  ← change this one value
  employee_id=43, address="Paris"

Parquet cannot open that file and patch the one record. The usual result is a replacement file:

Before: employees-part-08.parquet
After:  employees-part-08-v2.parquet

If the files form an analytical table, Iceberg or Delta Lake should publish that replacement through a new snapshot. Readers then see the entire old table or the entire corrected table, with a rollback option if the correction was wrong.

If the product changes employee records constantly, I would keep those records in an operational database. Parquet remains the analytical copy, where its columnar layout is an advantage rather than an obstacle.

Sources