Every Record Belongs to Exactly One Partition

In F01, we learned how distributing work across multiple workers can improve processing speed. Now let’s consider a different problem: How do we decide which records each worker should process? Suppose our nightly validation job processes 21,154 order events using 12 workers. To distribute the workload, we divide the dataset into 12 partitions. Each partition contains a subset of the records and can be processed independently by a worker.

But who decides which record belongs to which partition? That’s the responsibility of a partition function. A partition function examines a record and assigns it to a partition.

Although this sounds straightforward, incorrect partitioning can cause two serious problems:

1. 01Missing records: Some records are not assigned to any partition and are never processed.
2. 02Duplicate records: Some records are assigned to multiple partitions and get processed more than once.

For example, an incorrect partitioning rule could turn our 21,154 input records into 21,207 partitioned records. The result is that 53 records appear twice, leading to incorrect order counts.

Rules for Correct Partitioning

To prevent these problems, a partitioning strategy must satisfy two properties:
1. Completeness: Every record must be assigned to a valid partition.
2. Disjointness: Every record must belong to only one partition.

Together, these properties ensure that each input record is assigned to exactly one partition.

Why the Partition Function Matters

A well-defined partition function takes one record and returns exactly one valid partition number. This makes completeness and disjointness easier to guarantee. However, if partitions are created using separate filtering conditions, mistakes such as overlapping ranges or gaps between boundaries can lead to duplicate or missing records. We’ll also explore stable partitioning, which ensures that records with the same key are routed consistently across different runs.

Now that we understand why partitioning matters, let’s explore how records are assigned to different partitions.
We’ll use the same dataset of 21,154 order events and distribute it across 12 partitions, numbered from 0 to 11.

Consider the following order event:

{
  "event_id": "ev-000001",
  "delivery_id": "dl-27e4102e32",
  "supersedes_event_id": null,
  "order_id": "ord-00001",
  "customer_id": "cust-0134",
  "event_type": "order_placed",
  "amount_cents": 699,
  "event_time_utc": "2026-03-02T00:08:54.332Z",
  "ingest_time_utc": "2026-03-02T00:09:02.929Z"
}

How do we decide which partition should receive this record? There are several partitioning strategies. Let’s understand three common approaches.

1. Round-Robin Partitioning

Round-robin partitioning distributes records across partitions in sequence, without considering their values.
For example, with 12 partitions:

• Record 0 goes to Partition 0.
• Record 1 goes to Partition 1.
• Record 2 goes to Partition 2.
• ...
• Record 11 goes to Partition 11.
• Record 12 goes back to Partition 0.

The assignment follows a simple rule: Partition = Record position % Number of partitions

Here, % is the modulo operator, which returns the remainder after division.
For our first record: 0 % 12 = 0
Therefore, ev-000001 is assigned to Partition 0.

Round-robin is useful when we want to distribute records evenly. However, it doesn’t guarantee that records belonging to the same customer or order will stay together.

2. Hash Partitioning

Hash partitioning uses a record’s key to determine its partition. A key can be a field such as customer_id, order_id, or event_id.

Suppose we want all events belonging to the same customer to be processed together. We can partition records using customer_id.
The process works as follows:

1. 01Take the customer’s ID, such as cust-0134.
2. 02Pass it through a hash function, which converts the key into a numeric value.
3. 03Divide that number by the number of partitions and use the remainder to select a partition.

The formula is: Partition = Hash(key) % Number of partitions

Our reference router uses SHA-1 and converts the first eight bytes of the hash into an integer.

For cust-0134, the resulting value is: 2,381,997,355,534,491,765
Applying the partitioning rule: 2,381,997,355,534,491,765 % 12 = 9

Therefore, this event is assigned to Partition 9.

Why is this useful?
Every event belonging to cust-0134 produces the same hash value. As long as the hashing rule and partition count remain unchanged, all those events go to Partition 9. This is important for operations that need to process related records together, such as calculating customer-level metrics or maintaining customer history. However, hash partitioning doesn’t guarantee that every partition receives the same number of records. We’ll examine why shortly.

3. Range Partitioning

Range partitioning assigns records based on the range in which a key falls. For example, suppose we want to distribute order events using order_id. We can sort the order IDs and divide their values into 12 ordered ranges. Each partition is responsible for a particular range of order IDs. In our reference dataset, the first range ends just before ord-00367. Since ord-00001 falls within that first range, the record is assigned to Partition 0.

Unlike round-robin or hash partitioning, range partitioning keeps nearby key values together. This is useful when processing or retrieving records by ordered ranges, but the range boundaries must be chosen carefully to avoid uneven distribution.

Comparing the Three Strategies

Let’s apply these strategies to all 21,154 records and compare how evenly they distribute the data.

Our round-robin strategy assigns records using: Partition = Record position % 12

Choosing the Right Partitioning Strategy

Different partitioning strategies serve different purposes:

• Round-robin distributes records almost evenly but doesn’t keep related records together.
• Hash partitioning keeps records with the same key together, but partitions may contain different numbers of records.
• Range partitioning groups records by ordered key ranges, but its effectiveness depends on how those ranges are defined.

We’ll explore uneven partition sizes and their performance impact in F02.2.

Distributing records across partitions isn’t just about balancing the workload. We must also ensure that every record is assigned correctly and related records are grouped when necessary. As we discussed above, what is more important in terms of partitioning is its completeness, disjointness, and, if needed, keeping related records together.

What Makes a Good Partition Function?

A partition function should take one record as an input and return exactly one valid partition number as its output. For example: partition_id = partition_function(record)

If the function assigns every input record to exactly one valid partition, completeness and disjointness are guaranteed by design.

However, consider a different approach. Instead of using one function, a developer creates 12 independent filters, one for each partition. For example:

• Partition 0 accepts orders from ord-00001 through ord-00367.
• Partition 1 accepts orders from ord-00367 through ord-00733.

Notice that ord-00367 appears in both ranges. If both boundaries are inclusive, records belonging to ord-00367 will be assigned to both partitions. Similarly, gaps between ranges could cause some records to be excluded entirely. Unlike a single-valued partition function, independent filters don’t automatically guarantee completeness or disjointness.

Key Ownership: Keeping Related Records Together

Some distributed operations require records with the same key to be processed together. Suppose we’re building a complete order history for each customer. A customer may have placed several orders, and each order may generate multiple events. To process that customer’s history efficiently, we want all events belonging to the customer in the same partition. This property is called key ownership.

Hash partitioning by customer_id provides this guarantee because the same customer ID always produces the same partition number when the routing configuration is unchanged.

Now, let’s see what happens when we use different partitioning strategies. Below, we compare our reference dataset.

Here’s what the results tell us:

• Round-robin: Customer events are distributed independently of customer identity. In our dataset, every customer has events across multiple partitions.
• Hash by event_id: Each event is routed using its event ID, so events from the same customer can land in different partitions.
• Hash by order_id: All events belonging to the same order stay together, but one customer may have multiple orders assigned to different partitions.
• Hash by customer_id: All events belonging to the same customer are guaranteed to stay in one partition.

Hash partitioning keeps records with the same partitioning key together. For example, partitioning by order_id keeps all events for an order in the same partition, but events from different orders belonging to the same customer may end up in different partitions.

But does Hash Partitioning Guarantee Balanced Work?
Not necessarily. Hashing can distribute different keys across partitions, but it doesn’t account for how many records belong to each key. Some customers generate far more events than others. If several high-volume customers happen to map to the same partition, that partition can become much larger than the rest.

We have:

• 21,154 order events
• 797 customers
• 12 partitions
• The largest customer has 1,261 events, representing about 6% of all records.

Why Does This Matter?

Imagine that each partition is processed by a separate worker. A worker assigned 3,789 records may take significantly longer than one assigned only 926 records, assuming similar processing cost per record. Even if most workers finish quickly, the job may have to wait for the worker processing the largest partition. This problem is known as data skew, which we’ll explore in detail in F02.2.

Even when every record is assigned to exactly one partition, routing problems can still occur. Let’s examine two common mistakes involving hash partitioning.

1: Using an Unstable Hash Function

Suppose we’re building customer histories. All events belonging to the same customer must be assigned to the same partition so they can be processed alongside that customer’s existing state.
Let’s consider the partitioning strategy: partition = hash(customer_id) % 16

This looks reasonable. The hash function converts the customer ID into a number, and the modulo operation selects one of 16 partitions. But there’s a problem. Python’s built-in hash() uses randomised hashing for strings by default. Its result can change between independent Python processes.

A customer with ID cust-0001 is processed on two different days by separate Python processes. Both processes use: hash("cust-0001") % 16

How Can We Fix It?

Use a deterministic hash function, such as the SHA-1-based function in our reference router. A deterministic hash produces the same result for the same input, provided the hashing algorithm, key representation, and configuration remain unchanged. This ensures consistent routing across machines and program executions.

2: Changing the Partition Count

What About Doubling the Partitions?

When the partition count doubles from 16 to 32, approximately half the customers move.
For a given hash value, the new remainder is either the previous partition number or that number plus 16. This explains why doubling the partition count moves roughly 50% of the keys, which we verified by running it in our experiment.

Why Does This Matter?

Changing the partition count can affect systems that maintain data or processing state by partition. If a customer’s state is stored in its previous partition, moving new events to another partition without migrating the state may cause incorrect results. Systems that support repartitioning or rescaling need a strategy for redistributing data and state safely.

How can we tell when records are missing, duplicated, or assigned inconsistently?
We can run the checks:

1. 01Missing records: Some input records are not assigned to any partition.
2. 02Duplicate assignments: The same input record appears in multiple partition outputs.
3. 03Unexpected output counts: The sum of partition counts differs from the number of input records.
4. 04Inconsistent routing: The same key is assigned to different partitions across runs, even though the routing configuration was supposed to remain unchanged.

To identify the cause of these problems, examine how records are distributed across partitions and use automated checks to detect missing, duplicated, or incorrectly assigned records.

Choosing a Partitioning Strategy

Before choosing a partitioning strategy, consider these:

1. 01How will each record be assigned? Define a clear function that maps every input record to exactly one valid partition.
2. 02Which key should determine the partition? Choose a key that keeps together the records required by the computation.
3. 03Are the assignments complete and disjoint? Verify that no input record is missing or assigned more than once.
4. 04Is the routing deterministic? Ensure the same key maps to the same partition across processes and runs when the configuration is unchanged.
5. 05What happens when the partition count changes? Understand how many keys may move and whether existing data or state needs redistribution.

If the implementation uses independent filters instead of a single routing function, we should pay particular attention to overlapping boundaries and gaps.

Check Your Understanding: Does Partitioning by Order Keep Customers Together?

Suppose we partition the orders dataset into 12 partitions using a hash of order_id. A colleague argues:
“Every order belongs to a customer, so partitioning by order ID should also keep all of a customer’s events together.”
Is this correct?

Think about which key the hash function actually uses before revealing the answer.

Try It Yourself

When the F02 workbench is available, run:

The reference result shows Partition 7 containing 3,789 records.

Now change the partitioning key to event_id:

The largest partition now contains 1,867 records.

Compare the results and consider why changing the key affects both partition sizes and which records stay together.

Partitioning allows distributed systems to divide data into smaller pieces that can be processed independently.

In this lesson, we explored three important properties of partitioning:

1. 01Completeness: Every input record must be assigned to a partition.
2. 02Disjointness: No input record should be assigned to multiple partitions.
3. 03Key ownership: Records sharing a required key should be assigned to the same partition when the computation needs them together.

We also learned that:

• Incorrect range boundaries can duplicate records. Our example produced 21,207 assignments from 21,154 input records because 53 records appeared in two partitions.
• Hash partitioning doesn’t guarantee equal workloads. Hashing by customer_id produced a partition containing 3,789 records, more than twice the average.
• Routing must be deterministic when consistency matters. Python’s built-in string hash() may produce different results across processes.
• Changing partition counts can move many keys. Increasing from 16 to 17 partitions reassigned approximately 94% of customers in our reference dataset.

Sources. Capture: reference router 0.1, Python 3.13, 2026-10-13; fixture orders_s0_small.jsonl sha256 5ac4be13…, 21,154 records, 797 customers. The stable hash is SHA-1 of the UTF-8 key, first eight bytes as an integer; Python’s per-process string hashing is documented under PYTHONHASHSEED in the Python reference. The “every record exactly once” framing follows the partition-function definition used by MapReduce-style systems (one key, one reducer); the page’s figures are the module’s own.