Why adding more workers can make a job slower

On 16 single-core workers the nightly validation job took 175 seconds. On 64 it took 261 seconds: 86 seconds slower, with the same code and input data. Why did adding more workers make the job slower?

Let’s rebuild the job’s runtime on 16 and on 64 workers, one part at a time. Each part comes with the assumption it makes, so you can see exactly where the ideal stops.

How Runtime Changes with Worker Count

Notice what happens as we add workers:

• Parallel work decreases: More workers share the same amount of processing.
• Per-worker overhead increases: Every additional worker adds 3 seconds of overhead.
• Serial work and fixed overhead remain unchanged: Adding workers does not reduce these costs.

An important observation: Running the job with one worker on the cluster takes 1,285.4 seconds, while running it as a standalone process without the cluster takes only 1,242.4 seconds. The difference is 43 seconds of additional overhead. With only one worker, the cluster provides no parallel-processing benefit, but we still pay the cost of setting it up and managing that worker.

Our simulator incurs 3 seconds of overhead per worker, including launch, work assignment, and result collection.

Adding more workers can reduce processing time, but some parts of a job cannot be executed in parallel.

Let’s modify our nightly validation job to understand why. Suppose all order events arrive in a single compressed archive. Before workers can process the records, the coordinator must unpack the archive. In our example:

• Serial work: Unpacking the archive takes 150 seconds, followed by 5 seconds to generate the final summary. Total: 155 seconds.
• Parallel work: Validating all records takes 1,232.4 seconds on a single processor.

Without cluster overhead, the total time on one processor would be: 155 + 1,232.4 = 1,387.4 seconds

Now, imagine we could add unlimited workers without any startup or coordination overhead. As we add workers, the parallel processing time gets smaller and approaches zero. However, the 155 seconds of serial work remain unchanged, regardless of how many workers we add. Therefore, even with unlimited workers, the job cannot finish in less than 155 seconds. The maximum possible speedup is:

Maximum speedup = Single-process time ÷ Serial processing time = 1,387.4 ÷ 155 ≈ 8.95×

This is an example of Amdahl’s Law, which explains how the portion of a program that cannot be parallelised limits its maximum speedup. The serial fraction is the proportion of total processing time spent on work that cannot be divided among workers.

In our example: Serial fraction (s) = 155 ÷ 1,387.4 ≈ 0.112 (11.2%)

According to Amdahl’s Law, the theoretical maximum speedup with unlimited workers is:

Even with unlimited computing resources, the job cannot exceed this theoretical speedup unless we also reduce the serial work.

Our job requires 155 seconds of serial work and 1,232.4 seconds of parallel work.

What Happens When the Dataset Gets Larger?

Now suppose the number of order events triples, but the serial work remains fixed at 155 seconds. The parallel work increases from 1,232.4 seconds to: 1,232.4 × 3 = 3,697.2 seconds

The new single-process runtime becomes: 155 + 3,697.2 = 3,852.2 seconds

Let’s return to our nightly validation job. With 16 workers, the job completes in 175 seconds. The team now increases the cluster to 64 workers, expecting a significant improvement. But will four times as many workers make the job four times faster?

We’ve seen that adding workers has two opposite effects:

• Parallel processing time decreases because more workers share the workload.
• Worker overhead increases because each additional worker requires coordination.

This means there is a point where these two costs are balanced, giving us the shortest possible runtime for our model. Let’s find that point.

How Runtime Changes with Worker Count

Our simulator uses the following costs:

• Serial work + fixed overhead: 50 seconds
• Total parallel work: 1,232.4 seconds
• Overhead per worker: 3 seconds

For any number of workers, the runtime is: Total runtime = 50 + (3 × workers) + (1,232.4 ÷ workers)

Let’s compare different worker counts.

Core-seconds are calculated using the model’s assumption that every worker is billed for the entire job duration. Values are rounded.

Notice the pattern:

• From 1 to 20 workers, the total runtime decreases as parallel processing becomes faster.
• At 20 workers, the job reaches its shortest runtime of 171.6 seconds.
• Beyond 20 workers, the increasing overhead outweighs the savings from parallel processing, so the job becomes slower.

If we plot worker count against runtime, the curve falls initially, reaches a minimum, and then rises again. This produces a U-shaped curve.

Calculating the Optimal Worker Count

For this simplified model, we can estimate the worker count that minimises runtime using:

Substituting our values:

√(1,232.4 ÷ 3) ≈ 20.3 workers

Since we need a whole number of workers, 20 workers is the best choice for minimising runtime in this example. This formula assumes evenly distributed work and no additional resource bottlenecks. Real systems may require measurements to find their optimal configuration.

The operations team asks you to choose the worker count that completes the nightly job in the shortest time.

What About Resource Cost?

Let’s look at the last column of the table. A core-second represents one CPU core allocated for one second. In our billing model, every worker is charged for the entire job runtime, whether it is actively processing or waiting.

Compare two configurations:

Using 20 workers makes the job 56.5 seconds faster, but requires substantially more billing on compute resources.

Which configuration is better depends on the requirement:

• If finishing as quickly as possible matters most, choose 20 workers.
• If minimising compute cost matters more, fewer workers may be preferable, provided the job meets its deadline.

We’ll explore these cost and performance tradeoffs in F01.3.

Does the Optimal Worker Count Always Stay the Same?

No. The optimal number of workers changes with the workload and overhead:

• More parallel work: A larger workload can benefit from additional workers.
• Higher per-worker overhead: Expensive worker startup or coordination makes fewer workers more attractive.
• Shared-resource bottlenecks: Disk, network, or other resource limits may prevent additional workers from improving performance.

Our simulation demonstrates three important lessons:

1. 01More workers introduce additional overhead: Parallel processing becomes faster as workers are added, but the cost of managing those workers also increases. Beyond a certain point, adding workers can slow the job down.
2. 02Serial work limits maximum speedup: Tasks that cannot be parallelised take the same amount of time regardless of how many workers are available.
3. 03Distributed execution has a cost: Running a job on a cluster with just one worker can be slower than running it as a standalone process because of additional setup and coordination overhead.

Limitations of the Model

Our simulation makes a few simplifying assumptions:

• Sequential worker coordination: The coordinator starts workers and collects results one at a time. Real distributed engines can perform some of these operations concurrently.
• Equal workload distribution: Every worker receives the same amount of work. In F02, we’ll explore how uneven data distribution affects performance.
• No failures or retries: The model assumes every worker completes successfully. We’ll study failure recovery in F04.
• No additional resource bottlenecks: The optimal-worker formula assumes processing time decreases as work is divided. Shared disks or network limits can change the result, as we saw in F01.1.

Even when work is distributed evenly, not every worker finishes at the same time. A worker might run slower because of a busy disk, limited CPU resources, or contention with another process. In distributed processing, some stages require all workers to finish before the next stage can begin. This synchronisation point is called a barrier. The slowest worker determines when the stage finishes. Let’s see how this affects our nightly validation job.

With 16 workers, each worker normally completes its share of processing in 77 seconds. Now suppose one worker takes 30% longer than the others.

How Can We Reduce This Problem?

One possible approach is to divide the workload into smaller tasks, allowing work to be distributed more flexibly instead of assigning one large portion to each worker. We’ll explore workload distribution and uneven partitions in F02. Another issue worth investigating is hidden serial work. A job may contain an unexpected step that runs on only one process. Such a step can limit speedup even when the rest of the work is highly parallel.

How can you tell when adding workers is no longer helping? In our simulator, look for three warning signs:

1. 01Worker overhead grows faster than processing time decreases: Additional workers cost more time to manage than they save through parallel processing.
2. 02Total runtime increases despite faster parallel processing: Each worker finishes its share sooner, but the complete job takes longer.
3. 03Serial work limits speedup: A significant portion of runtime remains unchanged, regardless of how many workers are added.

In real distributed systems, similar symptoms include:

• A job becomes slower after increasing the cluster size.
• Worker startup or task scheduling takes a significant portion of total runtime.
• Processing speed stops improving despite adding more workers.
• Most workers finish early but must wait for one slow task.

Before increasing the number of workers, ask yourself four questions:

For our simplified model, the worker count that minimises runtime is approximately:

However, this formula assumes evenly divided work and no additional resource limits. Real systems may behave differently.

Let’s return to the nightly validation job. Previously, each worker introduced 3 seconds of overhead. Now suppose worker startup, coordination, and result collection become more expensive, increasing the overhead to 6 seconds per worker. The total parallel work remains unchanged at 1,232.4 seconds.

If per-worker overhead doubles from 3 to 6 seconds:

• What happens to the optimal worker count?
• How long will the job take with 16 workers?

Try It Yourself

When the F01 workbench is available, run:

The simulator compares worker counts and reports their execution times. Observe the worker count with the shortest runtime and identify the two competing costs: parallel processing time and per-worker overhead.

Adding more workers reduces the amount of parallel work each worker must perform, but it also introduces additional coordination overhead.

A distributed job’s total runtime depends on four components:

1. 01Serial work: Operations that cannot be divided among workers.
2. 02Fixed overhead: Setup and coordination costs paid once per job.
3. 03Parallel work: Processing time that decreases as work is distributed among more workers.
4. 04Per-worker overhead: Costs that increase with the number of workers.

In our nightly validation example:

• 16 workers completed the job in 175 seconds.
• 64 workers increased the runtime to 261.3 seconds, approximately 86 seconds slower.
• 20 workers provided the shortest runtime of 171.6 seconds in the original model.

We also learned that Amdahl’s Law limits the maximum speedup when part of a job cannot be parallelised. Additionally, a single slow worker can delay an entire processing stage when all workers must finish before execution continues.

Where This Matters

In production systems such as Apache Spark, similar performance problems can arise from driver-side processing, task scheduling overhead, uneven workloads, and workers waiting at stage boundaries.

We’ll explore Spark’s execution model and scheduling behavior in S01.

Sources. Capture: reference model sim 0.1, Python 3.13.16, 2026-10-13; profiles nightly_validate, lab_cluster and the page-only nightly_validate_unpack (150 s unpack step) and lab_cluster_c1x2 (6 s per worker). Amdahl, G. M. (1967), “Validity of the single processor approach to achieving large scale computing capabilities”, AFIPS Spring Joint Computer Conference, vol. 30, pp. 483–485. Gustafson, J. L. (1988), “Reevaluating Amdahl’s law”, Communications of the ACM 31(5), pp. 532–533.