Why can every worker be right and the report still be wrong?

The problem

Every worker calculated its share of revenue correctly, yet the final report showed the wrong numbers. The mistake wasn’t in any one worker. It was in how their results were combined.

01LessonsRead three lessons on merging partials, the cost of a shuffle and joins.Read

02WorkbenchBuild the two-stage simulator every later stage runs on.90–120 min

03TicketsDiagnose and repair five production tickets with your simulator.Repair

04Decision recordChoose the stages and join for the weekly customer report, today and at 3×.Decide

Check your early accessWe'll confirm your invite right after you sign in.

The idea, in four steps

01→Partial results

Each worker totals its own records into a partial result.

02→Exchange by key

→

Partials for the same key are sent to the same task, across the network.

03→Merge per key

Each key is merged in one task, so a big key makes a big task.

04One report

report

The merged totals become the report. It’s only right if every partial was combined the right way.

Combining results means moving them between workers. What moves, and where it lands, decides both the answer and the cost.

WorkPartial resultSent over the networkThe largest merge