Skip to content
ROOT
STUB

Partitioning

The pattern: split a dataset across N nodes so each handles only 1/N. Used for storage (sharding) and computation (parallel processing). Routing strategies: range (good for ranges; hot-spots), hash (uniform; bad for ranges), consistent-hash (resilient to node changes), explicit (Iceberg’s bucket/truncate transforms).

The trade-off: balance vs. flexibility vs. rebalance cost. Hash partitioning balances perfectly but kills range queries. Range partitioning supports ranges but hot-shards. Consistent-hash mitigates rebalance pain when nodes join/leave. The “hot shard” problem (one partition gets disproportionate traffic) shows up everywhere; mitigations are workload-specific.

[Deepen Year 2 Phase 8 with a 50-line consistent-hash ring implementation in Go.]

  • Replication — partitioning answers “where does the data live?”; replication answers “how many copies?”. You always pick both.
  • CAP and PACELC — partition (the network event) is named after this; partitioning (the data layout) determines how big the failure domain is.
  • Routing and addressing — same problem at the network layer: route packets to the right node.
  • Load balancing — consistent-hash LB is partitioning applied to request streams.
  • LSM vs. B-tree — within-shard layout choice; partitioning is between-shard.
  • Stream processing — Kafka partition keys are the parallelism axis for Flink jobs.

First touched in Year 2 Phase 8 via DDIA Ch. 6; reinforced in Year 3 Phase 15 (Iceberg partition spec) and Year 3 Phase 16 (Kafka topic partitions).