Write-Ahead Log (WAL)

Write the change to a sequential log before applying it to the data structure. The pattern under Postgres, Kafka, etcd, Iceberg snapshots, and every database that survives a crash.

Write the change to the log first. Then apply it to the data structure. Crash recovery: replay the log. Status: STUB — promoted to OUTLINE in Y2 Phase 9.

What this pattern is

A write-ahead log (WAL) captures every change before applying it to the durable data structure. The log is sequential (fast to write) and append-only (cheap to fsync). The data structure (B-tree, LSM, table file) can be updated lazily because the log is the source of truth. On crash, the database replays the log to bring the data structure back to the last consistent state.

The pattern separates what happened (the log) from how it’s stored (the data structure). The log is the primary durability contract. Once a log entry is fsynced to disk, the change is considered committed even if the corresponding data-structure update hasn’t been applied yet. Recovery becomes a matter of replaying the tail of the log from the last checkpoint. This design is why databases can survive kill -9, power failures, and OS crashes without losing committed writes.

This pattern is the foundation of durability for Postgres, MySQL InnoDB, SQLite, etcd, Kafka (the log is the database), Iceberg’s snapshots, and essentially every system that must survive a power-off without losing committed writes. It composes naturally with replication (ship the log to replicas), delivery-semantics (log offsets are how exactly-once-ish is built), and snapshot-plus-delta (the log is the delta).

Concrete instances in the wild

  • Postgres WAL. The pg_wal/ directory holds 16MB segments. Every INSERT, UPDATE, and DELETE writes to the WAL before touching data pages. pg_switch_wal() forces a segment rotation; pg_current_wal_lsn() shows the current position.
  • MySQL InnoDB redo log. Same pattern, different naming. ib_logfile0 and ib_logfile1 are the equivalent of Postgres’s WAL segments.
  • SQLite WAL mode. Before 2010, SQLite used rollback journals; after 2010, WAL mode became the default for concurrent read/write workloads. Same pattern applied at embedded scale.
  • etcd Raft log. etcd’s WAL is also its consensus log. Every state change gets logged before commit; recovery is log replay; replication is log shipping.
  • Kafka topics as WALs. Kafka took the pattern to its logical conclusion: the log is the primary abstraction, and consumers derive state by replaying it. There’s no separate data structure to keep in sync.
  • Iceberg manifest files. Each Iceberg commit produces a new manifest entry. The manifest chain is effectively a WAL over the table’s snapshot history.
  • RocksDB WAL. LSM engines still use WALs; the LSM’s memtable is the pending data structure, and the WAL survives crashes before memtable flush.
  • Filesystem journals. ext4, XFS, and NTFS all journal metadata changes before applying them. Same pattern, filesystem layer.
  • Application-level event sourcing. Systems that store events instead of state (accounting ledgers, banking transactions, audit logs) implement the WAL pattern at the application layer.

Why this pattern matters

Without a WAL, durability becomes a race. A traditional in-place update pattern writes to a data page and hopes the page hits disk before the power dies. If it doesn’t, you get a torn page: half the update is durable, half isn’t, and the database can’t reason about which. Some engines mitigate with double-write buffers or full-page images; all of them are trying to bolt some form of write-ahead-log semantics onto a design that wasn’t built for it.

With a WAL, durability becomes a property: once the log entry is fsynced, the change is committed. The data-structure update is bookkeeping. Recovery is deterministic: replay everything after the last checkpoint. Crash consistency is proven, not hoped for.

The pattern also enables replication. Ship the log to replicas; each replica applies the same operations in the same order and reaches the same state. Postgres streaming replication, MySQL binlog replication, Kafka’s follower protocol, etcd’s Raft — all use the WAL as the replication substrate. Without WAL, replication requires expensive full-state comparison or lockstep operation.

The 15-year arc of database engineering has been increasingly WAL-centric. Kafka took the pattern to its extreme: the log is the database. Iceberg applied the pattern to object storage. Event sourcing lifted it to the application layer. Every senior data engineer should be able to recognize the WAL pattern in any storage system they encounter.

Depth progression

STUB     ← you are here.
OUTLINE  Promoted when Y2 Phase 9 (SQL & databases) walks Postgres WAL internals.
DEEP     Promoted after Y4 when Kafka + Iceberg + Postgres are all operational —
         three instantiations of the same pattern on basecamp.

Preview: what OUTLINE will answer

When Y2 Phase 9 promotes this entry to OUTLINE, it will name:

  • PROBLEM. How do you guarantee durability of committed writes across crashes without paying the cost of syncing the full data structure on every write?
  • PRINCIPLES. Sequential log writes (fast fsync). Append-only (no in-place mutation of log). Log entry represents complete change intent. Data-structure application is idempotent replay. Checkpoints bound recovery time.
  • TRADE-OFFS. Sync commit vs async commit (latency vs durability). Log retention vs storage cost. Group commit (batching) vs single-transaction commit. WAL compression vs decompression CPU cost.
  • TOOLS (time-stamped as of 2026-06): Postgres WAL + pg_receivewal, MySQL binlog, SQLite WAL mode, Kafka topics, etcd Raft log, RocksDB WAL, filesystem journals (ext4/XFS).

The DEEP promotion, after Y4 when Kafka + Iceberg + Postgres are all operational, will add MASTERY (operating WAL-based systems for a year across three storage systems), COMPARE (Postgres WAL vs Kafka log vs Iceberg manifests), OPERATE (WAL-related incidents like disk-full, replication lag, checkpoint misconfiguration), and CONTRIBUTE (a Postgres or Kafka doc fix or minor improvement).

Canonical references

  • Bruce Momjian’s Postgres internals talks and slides — the definitive walkthrough of Postgres WAL mechanics. Free at momjian.us.
  • Martin Kleppmann, Designing Data-Intensive Applications — Chapter 3 covers log-structured storage; Chapter 5 covers replication protocols.
  • Jay Kreps, The Log: What every software engineer should know — LinkedIn engineering blog post, 2013. The canonical modern essay on why the log matters.
  • Apache Kafka: The Definitive Guide, chapters on internals and replication.
  • SQLite’s WAL mode documentation — a rare public reference implementation.

Cross-references