Why can more workers make a job slower?

The problem

A nightly data validation job took 175 seconds on 16 workers. The team moved it to 64, expecting it to finish sooner. It took 261 seconds — 86 seconds slower, with the same code and the same data.

01LessonsThree lessons on what sets a job’s runtime, and its bill.Read

02WorkbenchBuild worker-cost-sim, the simulator every later stage runs on.75–90 min

03TicketsDiagnose and repair four production tickets with your simulator.Repair

04Decision recordSize the fulfilment feed for today and two years of growth.Decide

Check your early accessWe'll confirm your invite right after you sign in.

The idea, in four steps

01→One large job

One process works through every record, one after another.

02→Split the work

Cut it into equal pieces.

03→Multiple workers

Each worker takes one piece, all at the same time.

04Combine results

Collecting each result takes a turn, and every extra worker adds another.

Splitting work can reduce processing time, but coordinating workers also has a cost.

WorkCoordination, one turn per worker