Skip to content

Lakehouse

What it is

A lakehouse combines low-cost object storage with table-management features expected by analytical databases. Amazon S3 is object storage: it stores objects, but it does not by itself define multi-file table transactions, isolation, or table rollback.

Apache Iceberg and Delta Lake are table formats that place a transactional table layer over data files, commonly Parquet. Iceberg uses versioned metadata and manifests; Delta Lake uses a versioned transaction log. Both make a collection of files behave as one table.

Why it matters

Without a table layer, a reader can observe a partially updated set of files while a writer is changing a dataset. A table format gives readers a consistent snapshot, supports ACID properties, allows concurrent writers to detect conflicts, and makes historical versions available for time travel and rollback.

How it works

Writers do not modify existing object files in place. They write replacement data files and prepare new metadata that points to them. The table then atomically commits a new snapshot or log version. Readers resolve the current table version from metadata and read only the files referenced by that version.

Writes normally use optimistic concurrency control: a writer plans against a current version, then validates and retries or fails if another compatible commit has not occurred. This avoids holding a long-lived global lock while data files are written.

Example

Assume snapshot 17 references A.parquet, B.parquet, and C.parquet. To change data held by B.parquet, a writer produces B2.parquet and commits snapshot 18, which references A.parquet, B2.parquet, and C.parquet. Readers on snapshot 17 continue to see B.parquet; new readers see B2.parquet. Rolling back selects an earlier valid snapshot, subject to metadata and file-retention policies.

Trade-offs and limitations

An update to one logical row can still rewrite a data file, so file sizing, partitioning, compaction, and write patterns matter. Transactional guarantees come from the table format and its catalog or log protocol, not from Parquet or S3 alone. Iceberg and Delta Lake have different metadata designs, integrations, and operational requirements.

Glossary

  • ACID: Atomicity, consistency, isolation, and durability; transaction properties that preserve correct table state.
  • Amazon S3: Amazon Simple Storage Service, an object-storage service.

Sources