Partitioning

Split data across nodes or files by a partition key. The pattern under distributed databases, Kafka topics, Iceberg tables, and Spark shuffles.

Split the data so each consumer reads less of it. The choice of partition key shapes everything downstream. Status: STUB — promoted to OUTLINE in Y3 Phase 21.

What this pattern is

Partitioning splits a dataset across nodes (for distributed databases) or across files (for table formats) by a partition key. The partition key is the load-bearing decision: a well-chosen key spreads load evenly, lets queries prune partitions they don’t need, and keeps related data co-located. A poorly-chosen key creates hot spots, forces full scans, or breaks join performance.

The same pattern shows up across the stack: Postgres partitioned tables, Cassandra’s partition keys, Kafka topic partitions (and ordering guarantees per partition), Iceberg’s hidden partitioning, Spark’s shuffle partitions, sharded caches. In every case, the same three questions decide whether partitioning works: What’s the partition key? How is data distributed across partitions given that key? What queries or operations benefit from the partitioning?

Partitioning is one of those patterns where getting the initial choice right matters disproportionately. Repartitioning a live system is expensive: you have to rewrite data, coordinate consumers, and often accept downtime. Teams that pick partition keys carefully at design time pay a small upfront cost; teams that pick keys casually pay a large repartitioning cost later, sometimes never actually paying it and living with the wrong design forever.

Partitioning interacts with replication (each partition is replicated independently), with consensus (each partition may have its own leader), and with load-balancing (the partition function is a form of load balancer).

Concrete instances in the wild

  • Postgres partitioned tables. Range, list, or hash partitioning. Common pattern: partition by created_at monthly for time-series data. Queries with WHERE created_at BETWEEN ... prune partitions.
  • Cassandra partition keys. Cassandra’s partition key determines which node the data lives on. All rows with the same partition key are co-located and can be efficiently scanned. Cross-partition queries are expensive.
  • Kafka topic partitions. A topic is split into N partitions. Producers hash the message key to pick a partition. Consumers read from specific partitions. Ordering is guaranteed only within a partition.
  • Iceberg hidden partitioning. Iceberg tracks partition transformations (day, month, bucket, truncate) as metadata. Queries with matching predicates prune partitions automatically. No explicit partition columns needed at write time.
  • Elasticsearch shards. Each index is split into shards; documents are routed to shards by document ID (or a custom routing key). Aggregations parallelize across shards.
  • DynamoDB partition keys. Each item has a partition key that determines physical placement. The same partition key means same physical partition means efficient point queries.
  • Spark shuffle partitions. Distributed joins and aggregations shuffle data across partitions. The number of shuffle partitions determines parallelism.
  • Sharded caches. Redis Cluster hash-slots, Memcached consistent hashing. Client-side partitioning distributes keys across cache nodes.
  • CockroachDB range partitioning. Automatic range-based sharding of tables; ranges split and rebalance dynamically.

Why this pattern matters

Without partitioning, distributed systems can’t scale beyond one machine’s capacity. Every data-heavy system that has ever grown beyond a single node has done so through some form of partitioning. Understanding the pattern means understanding how horizontal scaling actually works: not “add more machines and hope,” but “split the data such that each machine handles its share independently.”

Partition-key choice is where most partitioning goes wrong. A key that clusters (many rows with the same key value) creates hot partitions. A key that’s too fine-grained (unique per row) makes partition pruning useless. A key that doesn’t match query patterns forces every query to scan all partitions. A key that changes over time (using created_at when queries are actually by updated_at) creates cold storage of hot data. Getting this right requires knowing both the data distribution and the query patterns.

The pattern also drives consistency semantics. Kafka guarantees ordering within a partition but not across. Cassandra guarantees efficient co-located reads within a partition key but expensive cross-partition scans. Postgres partitioned tables lose transactional guarantees across partitions in some configurations. Each system’s partitioning model determines what queries can be efficient and what invariants can be enforced. A senior engineer designing on any of these systems understands the partition model first, then designs the query patterns around it.

Repartitioning is the failure mode that teaches humility. When a partition key stops working (data grew past the key’s original distribution, query patterns shifted, hot spots emerged), fixing it requires rewriting data. Kafka topic repartitioning: create a new topic, dual-write, cutover, decommission. Iceberg partition evolution: define new partition spec, backfill or accept mixed spec, retire old spec. Cassandra: worse; often just rewrite. Every experienced data engineer has a story about a bad partition key that lived for years past its usefulness because repartitioning was too painful.

Depth progression

STUB     ← you are here.
OUTLINE  Promoted when Y3 Phase 21 (DDIA Ch 6) introduces partitioning at the
         distributed-systems level; Y4 Phase 31 reinforces at the data layer.
DEEP     Promoted after Y4 end with at least one rebalance/repartition event
         on basecamp (Kafka or Iceberg) and observed query-performance impact.

Preview: what OUTLINE will answer

When Y3 Phase 21 promotes this entry to OUTLINE, it will name:

  • PROBLEM. How do you split a dataset across nodes or files such that queries stay efficient, load stays balanced, and the system can scale beyond a single machine?
  • PRINCIPLES. Partition key drives data placement. Query pruning depends on the key matching query predicates. Balanced distribution requires either uniform key values or a hash function. Repartitioning is expensive; get the initial choice right.
  • TRADE-OFFS. Hash partitioning (uniform distribution, no range queries) vs range partitioning (efficient range queries, hotspot risk). High-cardinality keys (fine granularity, more partitions to manage) vs low-cardinality (coarse granularity, hotspot risk). Static partitioning (declared at design time) vs dynamic (auto-split/merge). Composite keys (finer control, more complexity) vs single-column keys (simpler, less flexible).
  • TOOLS (time-stamped as of 2026-06): Postgres partitioned tables, Cassandra partition keys, Kafka topic partitions, Iceberg hidden partitioning, DynamoDB partition keys, CockroachDB ranges, Elasticsearch shards, Spark shuffle partitions.

The DEEP promotion, after Y4 with a real repartition event, will add MASTERY (operating a partitioned system through growth and a repartitioning), COMPARE (Kafka partitioning vs Iceberg partitioning vs Postgres partitioning), OPERATE (a real partition-key mistake and its consequences), and CONTRIBUTE (a documentation contribution or blog post about the partition-key decision).

Canonical references

  • Martin Kleppmann, Designing Data-Intensive Applications, Chapter 6 — the definitive modern treatment of partitioning.
  • Google’s Bigtable paper — the partition-by-row-key design that influenced modern NoSQL.
  • Apache Kafka documentation on topic partitioning and consumer group rebalancing.
  • Netflix’s Iceberg posts on hidden partitioning — the modern lakehouse approach.
  • Postgres partitioning documentation — declarative range and list partitioning at the SQL layer.
  • Cassandra: The Definitive Guide, chapters on partition-key design.

Cross-references