Distributed data processing
Partitions, shuffles, clocks, state, delivery guarantees, and the incidents they cause. Each lesson starts from a failure a real pipeline produces, shows the mechanism behind it, and ends with a fix and its cost.
Three engines, one set of mechanisms. Learn the mechanism once in plain Python, and Spark, Kafka and Flink each become a version of something you already understand.
The revenue dashboard reported 9.6% more revenue than expected. The source data was correct, but one publish task had written its results twice.
Already know this?Take the F01 diagnostic · 40 min →Early access
Save your progress so you don’t lose it.Save my progress →
The route
- Foundations trunk · 12 lessons live
- Kafka · no Spark needed
- Streaming
- Spark Batch
- Flink line · advanced
- Integration
Point at or tap a station to see its module.
Take the trunk, then the line for your engine. A passed module, its 40-minute diagnostic or a passed gate opens the next station.
How a module works
- LessonsPredict, then read the mechanism. Three per module, 13–43 min each.Not graded
- WorkbenchBuild a small simulator in Python, checked against a result oracle.Not graded
- TicketsDiagnose and repair failures you haven’t seen; your regression test must fail on the original.Graded
- Decision recordAt least two options, priced with your own numbers, every guarantee with its scope named.Graded · published rubric
A module is passed on evidence, not attendance.
Already know a module? Its diagnostic – 7 predictions and one unseen ticket, 40 minutes – opens the next one without taking it.
Track F · Foundations · 4 modulesLessons liveHands-on in early access
Distributed computing foundations
Build engine-independent mental models using small deterministic Python experiments before introducing Spark.
Parallelism and the economics of distribution
You can Predict when additional workers help or hurt and justify using a cost model
- F01.1What determines a job’s runtime?Sixteen processes, one disk: why the hourly job stopped speeding up at 63 seconds13 min
- F01.2Why adding more workers can make a job slowerIncreasing workers from 16 to 64 made the nightly job 86 seconds slower.20 min
- F01.3Balancing job performance and costWhy eight workers met the latency requirement while sixty-four cost nine times more and still finished late.18 min
Hands-on Early access
Workbench L-F01 worker-cost-sim · 5 tickets (one observe-only) · decision record “The fulfilment feed” Check your early access →
Already know F01?Module diagnostic · 7 predictions and one unseen ticket · 40 min · opens F02 →Early access
Gate G-FEarly access
Prove Track F in one gate
16 predictions (30 min), three tickets from the gate pack (about 40 min each) and one decision record (about 60 min), within seven days. Pass it and the Spark, Kafka and Streaming lines open without taking the four modules.
See what G-F asks →Coming next · 20 modules · Planned
Before you start
- You need
- Python functions, lists and dictionaries, and a test; SQL joins and grouping (the SQL course’s first three stages); reading a file from the command line.
- You don’t need
- Spark, Kafka, Flink, a cloud account, Kubernetes or containers to begin.
Where to start
- Junior
- Uses a framework's defaults; unaware of shuffles, partitioning, or batch/stream trade-offs. Start at F01.1 →
- Mid
- Understands partitioning and shuffles; writes batch and basic streaming jobs but tunes reactively. Take the four module diagnostics and start at the first one you miss, then your engine’s line →
- Senior
- Reasons about distribution, state, and exactly-once vs at-least-once; designs for skew, late data, and recovery. Take G-F, then the Integration line →
A level describes your work on this area, never you. No report yet? Take the F01 diagnostic →
Start with the failure you’ll meet first.
Start F01.1 · 13 min →15 scenarios to practise this area
Curriculum plan v2.0 · 13 Oct 2026 · 24 modules · 6 tracks · 7 gates · Read the full plan →
Live public · Early access invited members · Planned not written yet