Why can one partition slow down the whole job?

The problem

Uneven data distribution pushed the nightly job from 195 to 307 seconds. Every record was processed. Most workers simply finished early, then waited for one.

01LessonsThree lessons on partitions, skew and where the data sits.Read

02WorkbenchBuild partition-router, the tool every later stage runs on.75–90 min

03TicketsDiagnose and repair four production tickets with your router.Repair

04Decision recordKey and size the one-file-per-customer job for today and growth.Decide

Check your early accessWe'll confirm your invite right after you sign in.

The idea, in three steps

01→Records with keys

Every record carries a key, such as the customer it belongs to.

02→Grouped by key

Records with the same key land in the same partition, and some keys are much bigger.

03One task each

Small partitions finish early. Their workers wait for the largest, and they are still billed.

Grouping records by key keeps related data together, but the largest group decides when everyone is done.

WorkThe largest shareWaiting, still billed