Why One Partition Can Slow Down the Entire Job

In F02.1, we learned how to distribute records across partitions using round-robin, hash, and range partitioning. But correctly assigning records to partitions doesn’t guarantee that the workload is evenly distributed.

Let’s return to our nightly validation job. The team now needs to perform customer-level checks, which require all events belonging to the same customer to be processed together. To achieve this, they use hash partitioning by customer_id, distributing the records across 12 partitions processed by 12 workers. Surprisingly, the job takes longer to complete. For comparison, the team runs the same workload using hash partitioning by event_id, keeping the input data, worker count, and processing costs unchanged. The results are significantly different:

That’s a difference of 112 seconds, even though both configurations use the same computing resources.

Why Does the Partitioning Key Make Such a Difference?
In F01, we assumed that work was divided equally among workers. However, real datasets rarely have perfectly balanced distributions. Some customers generate far more events than others. When we partition by customer_id, all events belonging to a customer must go to the same partition. As a result, some partitions receive significantly more records than others. Eventually taking longer times to finish. Thus, the overall processing phase is dictated by the slowest task.
This uneven distribution is called data skew.

Why Does This Matter?
Imagine 11 workers have finished processing their assigned records, but the final worker is still running. If the next stage requires all partitions to finish, the entire job must wait for that last worker. Adding more workers won’t necessarily solve the problem if the underlying data remains unevenly distributed.

In this lesson, we’ll learn how to:

1. 01Measure data skew across partitions.
2. 02Understand why one slow task can delay an entire job.
3. 03Explore how partition count affects workload balancing.
4. 04Handle high-volume keys without violating key ownership requirements.

In F02.1, we distributed 21,154 order events across 12 partitions using customer_id as the partitioning key. The results showed that Partition 7 contained 3,789 records, significantly more than the other partitions. Let’s understand how this uneven distribution affects the job’s runtime.

Step 1: Calculate Each Partition’s Share

A partition’s share tells us what percentage of the total records it contains.
For Partition 7: Partition share = (Records in partition ÷ Total records) × 100

3,789 ÷ 21,154 × 100 ≈ 17.9%

This means Partition 7 contains approximately 17.9% of all records, even though we have 12 partitions. If the data were perfectly balanced, each partition would contain:

100 ÷ 12 ≈ 8.33%

Partition 7 therefore has more than twice the average share of records.

Step 2: Convert Record Counts into Processing Time

Our nightly validation job processes 60 million events. In this simulator, we assume the larger dataset follows the same partition distribution as our smaller fixture. We also assume that every record requires approximately the same processing time. From F01, processing one record takes an estimated 20.54 µs, including CPU validation and local disk reading.

The total processing work is:

60,000,000 × 20.54 µs = 1,232.4 seconds

This represents the total work if one processor handled every record. Since Partition 7 contains 17.9% of the records, its processing time is approximately:

1,232.4 × 0.179 ≈ 220.7 seconds

The simulator also adds 0.5 seconds of overhead per task for launching and reporting results.
Therefore: Partition 7 task duration = 220.7 + 0.5 = 221.2 seconds

For comparison, a perfectly balanced partition would take:

1,232.4 ÷ 12 + 0.5 = 103.2 seconds

Partition 7 takes more than twice as long as a balanced partition.

Partition 7 contains 17.9% of all records. The job requires 1,232.4 seconds of total parallel processing work, and each task has 0.5 seconds of overhead.

Step 3: Understand How One Slow Task Affects the Entire Job

We have 12 partitions and 12 workers, so each worker processes one partition. Some workers finish quickly, but Partition 7 takes 221.2 seconds. If the next processing stage must wait for all tasks to complete, the entire phase cannot finish until Partition 7 finishes. This makes Partition 7 the straggler, or the task that finishes last and delays the rest of the job. Now let’s calculate the total runtime. From F01, our cluster has the following overhead:

Therefore:

10 + 40 + 36 + 221.2 = 307.2 seconds

The other 11 workers finish their tasks earlier and must wait for Partition 7. In our simulator, this produces approximately 1,416 idle core-seconds during the parallel phase.

Why Does This Matter?

Compare the same job using different partitioning keys:

• Hash by event_id: Approximately 195.3 seconds.
• Hash by customer_id: Approximately 307.2 seconds.

The customer-based partitioning takes about 112 seconds longer because its largest partition contains significantly more records. However, we cannot simply switch to event_id if the job needs all events belonging to the same customer to stay together. We need a strategy that preserves key ownership while reducing the impact of uneven workloads. We’ll explore those options in the next sections.

In the previous section, we saw how one partition containing more records than others can significantly increase a job’s runtime. But how do we measure how unevenly the data is distributed? One useful metric is the skew factor.

What Is the Skew Factor?

The skew factor compares the largest partition with the average partition size.
Skew factor = Largest partition size ÷ Average partition size

For our dataset:

• Total records: 21,154
• Number of partitions: 12
• Average partition size: 21,154 ÷ 12 ≈ 1,763 records
• Largest partition: 3,789 records when hashing by customer_id

Therefore: Skew factor = 3,789 ÷ 1,763 ≈ 2.15

A skew factor of 2.15 means the largest partition contains approximately 2.15 times as many records as an average partition. A value close to 1 indicates a more balanced distribution. Higher values indicate greater imbalance.

Comparing Different Partitioning Strategies
Let’s compare how three partitioning strategies affect our nightly validation job.

Notice the difference:

1. 01Round-robin distributes records almost perfectly evenly, resulting in the shortest runtime.
2. 02Hashing by event_id also produces a relatively balanced workload because events have distinct IDs.
3. 03Hashing by customer_id creates a much larger partition, causing one task to run significantly longer.

However, remember that our customer-level processing requirement needs all events from the same customer to stay together. Round-robin and hashing by event_id do not guarantee this.
The fastest partitioning strategy is not necessarily the correct one for the computation.

How One Slow Task Delays the Job
Let’s examine how long each partition takes when using customer_id as the partitioning key. Our simulator assigns 12 partitions to 12 workers, with one task per worker.

Most tasks finish between 54 and 124 seconds, but Partition 7 takes 221.2 seconds. Why is Partition 7 so much slower?
It contains several high-volume customers:

• cust-0001: 1,261 events
• cust-0002: 904 events
• Another 69 customers assigned to the same partition

Together, they account for 3,789 records. Since all records within a partition are processed by one task, Partition 7 requires much more processing time. This slow-running task is called a straggler.

Our job runs 12 tasks on 12 workers. One task takes 221 seconds, while the remaining 11 tasks finish between 54 and 124 seconds.

We’ve seen that hashing by customer_id creates uneven partitions, with one task taking considerably longer than the others. But the job requires key ownership, so we cannot simply switch to round-robin partitioning. The team proposes another solution: What if we increase the number of partitions without increasing the number of workers?
Currently, we have:

• 12 workers
• 12 partitions
• One task per partition
• Total runtime: 307.2 seconds

Now suppose we increase the partition count to 24, while keeping the worker count at 12. Instead of processing one large partition, workers can process smaller tasks. When a worker finishes its current task, it can take another available task. This can improve workload balancing because faster workers don’t necessarily have to remain idle while other tasks are still running.

We increase the partition count from 12 to 24, but keep the same 12 workers and continue hashing by customer_id.

Why Does Performance Improve Even Though Skew Increases?

The skew factor compares the largest partition with the average partition size. When we increase the number of partitions, the average partition becomes smaller. However, all records belonging to one customer must still remain together. This means a high-volume customer can continue creating a relatively large partition, even as the average partition size decreases. As a result, the skew factor can increase. So why does the job finish faster? Because we now have more tasks than workers. Instead of each worker being assigned one partition for the entire phase:

1. 01A worker receives an available task.
2. 02It processes that task.
3. 03When it finishes, it takes another available task.
4. 04Processing continues until all tasks are completed.

This allows workers that finish smaller tasks to take on additional work. In our 24-partition simulation, the final worker completes its assigned work after 169.8 seconds, compared with 221.2 seconds in the original configuration.

The total runtime becomes:
10 s serial work + 40 s fixed overhead + 36 s worker coordination + 169.8 s processing phase = 255.8 seconds

Does Increasing Partitions Always Improve Performance?
Let’s revisit the results:

• 12 partitions: 307.2 seconds
• 24 partitions: 255.8 seconds
• 48 partitions: 214.5 seconds
• 96 partitions: 227.2 seconds
• 192 partitions: 234.7 seconds

The job becomes faster as we increase partitions from 12 to 48. However, beyond 48 partitions, performance starts to decline. This is because, Every task introduces overhead. In our simulator, each task requires an additional 0.5 seconds for launching and reporting results. As we create more partitions, we also create more tasks, increasing this overhead. With 192 partitions and 12 workers, there are 16 tasks per worker on average. More tasks also mean that workers may process several tasks sequentially. Therefore, when partitions outnumber workers, the processing phase depends on each worker’s accumulated task durations, not simply the duration of the single largest task. Eventually, the overhead and scheduling costs outweigh the benefits of using smaller tasks.

Finding the Right Partition Count
For this workload, 48 partitions provide the shortest runtime among the tested configurations: 214.5 seconds. This does not mean 48 partitions will be optimal for every dataset or cluster.

The best partition count depends on:

• How evenly records are distributed across keys.
• How much work each task performs.
• How many workers are available.
• How tasks are scheduled.
• How much overhead each additional task introduces.

In production, engineers should compare configurations using actual task durations, scheduling metrics, and overall job runtime.

We’ve seen that increasing the number of partitions can improve workload balancing. However, there is another approach: choosing a better partitioning strategy based on what the job actually needs. Before making changes, ask one important question: Does the job require all records belonging to the same key to stay together?
The answer to this determines which optimisation we can safely apply.

Strategies for Handling Data Skew

1: Change the Partitioning Key
If records can be processed independently, we don’t need to keep all events from the same customer in one partition. For example, our nightly validation job checks each event individually. It doesn’t need the complete history of a customer to validate an event. Instead of hashing by customer_id, we can hash by event_id. Our simulator shows:

Hashing by event_id distributes records more evenly because each event has its own identifier. The result is a 112-second reduction in runtime, along with lower computing costs. However, this solution is valid only when records can be processed independently. If the job needs customer-level history or state, changing the partitioning key would violate that requirement.

2: Isolate High-Volume Keys
Now suppose our job must process all events belonging to the same customer together. In this case, we cannot simply switch to event_id. Instead, we can identify customers generating unusually large numbers of events and assign them dedicated partitions. This technique is called hot-key isolation. In our dataset, the two largest customers are:

• cust-0001: 1,261 events.
• cust-0002: 904 events.

Previously, both customers were assigned to Partition 7, along with 69 other customers. This created a large partition containing 3,789 records. Let’s try isolating cust-0001.

We keep the total number of partitions at 12:

1. 01Assign all events from cust-0001 to one dedicated partition.
2. 02Hash the remaining 796 customers across the other 11 partitions.
3. 03Keep all events belonging to each customer together.

We isolate cust-0001, which contributes approximately 6% of all records, and distribute the remaining customers across 11 partitions.

Understanding the Hot-Key Processing Limit

Hot-key isolation improves workload distribution, but it cannot eliminate the work required to process a high-volume customer. Consider cust-0001. This customer has 1,261 events out of 21,154, representing approximately 6% of the dataset. Our nightly job requires 1,232.4 seconds of total parallel processing work. Therefore, processing this customer’s events requires approximately:
1,232.4 × (1,261 ÷ 21,154) ≈ 73.5 seconds

Including the 0.5-second task overhead, the task requires: 73.5 + 0.5 = 74.0 seconds

This creates what we call a key floor. A key floor is the minimum time required to process the records belonging to a particular key when all of that key’s work must be handled by one task. Even if we add more workers, the task processing cust-0001 still requires approximately 74 seconds.

Why Can’t More Workers Solve This?

Let’s isolate the three largest customers and compare configurations with 32 and 64 workers.

Notice what happens:

• With 32 workers, the parallel phase takes 74 seconds.
• With 64 workers, the parallel phase still takes 74 seconds.
• However, total runtime increases from 220 to 316 seconds because managing additional workers adds overhead.

The bottleneck is no longer the total number of workers. It is the amount of work that must remain together for a single customer.

Can We Process a Hot Key Faster?

Potentially, yes. If the computation allows it, we could divide one customer’s records into smaller groups, process them in parallel, and combine the partial results afterward. However, not every operation can be split safely. Some computations depend on processing events together or maintaining ordered state. We’ll explore how distributed aggregation and data movement can support such techniques in F03.

Our experiments demonstrate four important lessons about data distribution and processing performance.

1. 01Uneven partitions create uneven workloads: When records have similar processing costs, a partition’s share of the records determines how much processing work it receives.
2. 02The slowest worker can delay the entire job: When a processing phase requires all tasks to finish, one long-running task can determine when the phase completes.
3. 03More partitions can improve workload balancing: Allowing workers to process multiple smaller tasks can reduce idle time. However, too many tasks introduce additional overhead.
4. 04Hot keys can limit parallelism: If all records belonging to one key must be processed together in a single task, that task creates a minimum processing time that adding workers alone cannot reduce.

Limitations of the Model

Our simulator makes several simplifying assumptions:

• Local data access: Every worker reads from its own local disk. In F02.3, we’ll explore how reading data from other machines affects performance.
• Simplified task scheduling: Tasks are assigned in partition order to the first available worker. Real distributed processing engines use more sophisticated scheduling strategies.
• Fixed processing cost per record: We assume records require similar amounts of processing work. Real workloads may have records that differ significantly in size or complexity.
• No failures or retries: Every task completes successfully. We’ll explore task failures, retries, and recovery in F04.
• No splitting of individual keys: Each customer’s records remain together in one task. In F03, we’ll explore when work associated with a key can be divided and combined safely.

Why Measuring Skew Matters

The skew factor depends on the partitioning key, the number of partitions, and the actual distribution of records. For example, hashing by customer_id produces different skew factors depending on the number of partitions:

Increasing the partition count does not guarantee a lower skew factor. This is because changing the partition count changes how keys are assigned, and high-volume customers may end up together in the same partition. Furthermore, the skew factor alone doesn’t determine total runtime. Task scheduling, overhead, and the number of available workers also matter.

We’ve seen that increasing the number of partitions and isolating high-volume keys can improve performance. However, these techniques come with their own tradeoffs. Let’s examine three common mistakes.

Mistake 1: Assuming More Partitions Always Improve Performance

Increasing the number of partitions can distribute work more effectively, but every additional partition also creates a task that needs to be scheduled and executed. In our simulator, each task introduces 0.5 seconds of overhead for launching and reporting results.
As the number of tasks increases, this overhead can become significant.

Suppose we have:

• 192 tasks
• 12 workers
• 0.5 seconds of overhead per task

Mistake 2: Starting Large Tasks Too Late

Task scheduling order also affects job completion time. Imagine a worker receives a large partition only after processing several smaller partitions. Even if other workers have already finished, the job must wait for this large task to complete. In our simulator, tasks are scheduled in partition order. When we isolate high-volume customers, their partitions are placed first so those larger tasks can begin earlier. Starting large tasks earlier can reduce the chance that they become the final unfinished work.
However, real distributed processing engines use more sophisticated scheduling strategies, and the best scheduling order depends on task sizes, resource availability, and dependencies.

Mistake 3: Assuming Equal Record Counts Mean Equal Work

So far, our simulator assumes every record is approximately 270 bytes and requires similar processing time. Under this assumption, partitions containing similar numbers of records also require similar amounts of work. But real datasets are often more complicated.

For example:

• One partition might contain 10,000 small records.
• Another might contain 10,000 records with large nested JSON objects.
• A third might contain records requiring expensive transformations or external lookups.

Although the record counts are equal, the processing costs can be very different. This means a partition can be balanced by record count but still be skewed by data size or computational complexity.

How can we recognise when uneven data distribution is affecting performance?
In our simulator, look for four warning signs:

1. 01High skew factor: The largest partition contains significantly more records than the average. For example, a skew factor of 2.15 means the largest partition is more than twice the average size.
2. 02One task takes much longer: Most tasks finish quickly, but one or a few continue running for significantly longer.
3. 03High worker idle time: Several workers finish their tasks and remain idle while waiting for the remaining tasks.
4. 04Runtime exceeds the balanced estimate: The job takes substantially longer than expected when the total processing work is divided evenly.

What Should You Investigate?

When you notice these symptoms, start by examining:

• Partition sizes: How many records or bytes does each partition contain?
• Key distribution: Are a few high-volume keys responsible for most of the data?
• Task durations: How much longer do the slowest tasks take compared with typical tasks?
• Processing requirements: Must all records belonging to a key remain together?

For example, in our customer-partitioned job, Partition 7 contains 3,789 records, compared with an average of approximately 1,763. That uneven distribution explains why its task takes 221.2 seconds, while a balanced task would take approximately 103.2 seconds. These measurements help us identify the cause before changing the partition count or adding workers.

Before changing the partitioning strategy or increasing the worker count, ask yourself five questions:

1. 01Does the computation require key ownership? Determine whether records sharing the same customer, order, or other key must remain together.
2. 02How much work belongs to the hottest key? Estimate the processing time required for that key when its records must stay in one task.
3. 03How unevenly is data distributed? Measure partition sizes and the skew factor using the actual workload.
4. 04Would more partitions improve scheduling? Compare task sizes, tasks per worker, and the overhead introduced by additional tasks.
5. 05Can isolating hot keys improve the distribution? Identify high-volume keys, assign dedicated partitions where appropriate, and measure the resulting workload.

One important distinction: If records can be processed independently, choosing a more evenly distributed key may be sufficient. If the computation requires key ownership, we must preserve that requirement while exploring alternatives such as hot-key isolation or splitting computations that can safely be combined later.

Let’s revisit our customer-partitioned nightly validation job. The largest customer, cust-0001, contributes approximately 6% of all events. Our job requires 1,232.4 seconds of total parallel processing work. Because all events for this customer must remain together in a single task, processing them requires approximately 1,232.4 × 0.06 ≈ 73.5 seconds

Including task overhead, the minimum duration is approximately 74 seconds. Now suppose we increase the number of workers to 64.

Can We Go Below This Limit?

Potentially, but doing so requires changing how the computation handles a customer’s records. For example, if the operation supports combining partial results, we might split one customer’s events into smaller groups, process those groups in parallel, and merge their results afterward. However, this introduces data movement and may not work for operations that require strict ordering or shared state.

We’ll explore distributed aggregation and these tradeoffs in F03.

Try It Yourself

When the F02 workbench is available, run:

The reference result should show:

• Skew factor: 2.15
• Longest task: 221.2 seconds
• Total runtime: 307.2 seconds

Now isolate the largest customer by adding: --isolate-top 1 --partitions 11

Here, 11 partitions are used for the remaining customers, alongside one dedicated partition for the isolated customer. Compare the results and identify which partition now takes the longest.

Note: The workbench is planned. These commands are intended for use once it becomes available.

Distributed processing performance depends not just on the number of workers, but also on how evenly the workload is distributed across tasks.
In this lesson, we learned that:

1. 01Data skew creates uneven workloads: When some partitions contain significantly more work, their tasks take longer to complete.
2. 02The slowest task can delay an entire processing phase: Other workers may finish early but must wait for the remaining task.
3. 03More partitions can improve workload balancing: Workers can process multiple smaller tasks, reducing idle time. However, too many tasks introduce additional overhead.
4. 04Hot-key isolation can reduce skew: Assigning high-volume keys to dedicated partitions may prevent several heavy keys from overwhelming the same worker.
5. 05A hot key can limit parallelism: When all records for one key must remain in a single task, adding workers alone cannot reduce that task’s processing time.

In our nightly validation example:

• Hashing by customer_id produced a skew factor of 2.15, increasing runtime to 307.2 seconds.
• Increasing partitions from 12 to 48 reduced runtime to 214.5 seconds in the simulator.
• Isolating the largest customer reduced runtime to 238.8 seconds while preserving customer-level ownership.
• The hottest customer’s workload created a 74-second minimum processing phase under the single-task-per-key requirement.

Sources. Capture: reference router 0.1, Python 3.13, 2026-10-13; fixture orders_s0_small.jsonl sha256 5ac4be13…; profiles nightly_validate, lab_cluster (learner pack). The greedy list schedule is Graham’s list scheduling (R. L. Graham, “Bounds on multiprocessing timing anomalies”, SIAM J. Appl. Math. 17(2), 1969), whose makespan is within a factor of two of optimal; the page’s numbers are the module’s own.