Why can a small failure delay a whole job?

The problem

A worker failed during a three-second task, yet the nightly job took 104 seconds longer. On another night, one retried task wrote its results twice, and revenue came out 9.6% too high.

01LessonsRead three lessons on lost outputs, retried writes and retry-safe tests.Read

02WorkbenchBuild a coordinator whose workers fail, and prove its sink is right.90–120 min

03TicketsDiagnose and repair five production tickets with your coordinator.Repair

04Decision recordChoose the sink, mail path and recovery for the nightly ledger publish.Decide

Check your early accessWe'll confirm your invite right after you sign in.

The idea, in four steps

01→Tasks run

Workers run tasks, and later tasks need earlier tasks’ results.

02→A worker fails

A worker dies mid-task, and the results it was holding go with it.

03→Run it again

Whatever was lost runs again, and the job ends when the rerun does.

04Write once

result

second write

A retry may write its result again. The output must keep one copy, not two.

A failure costs whatever has to run again, and a retry must not write its result twice.

WorkFailureLost workRun againDuplicate write