Distributed Data Processing

A module is passed on evidence, not attendance: Stage 3 hidden tests and the Stage 4 rubric define a module pass (§8); track gates certify tracks. Required edges are satisfied at module granularity.

Basic Python (functions, lists/dictionaries, tests), SQL joins/grouping, structured file I/O, command-line execution, and basic reasoning about latency vs throughput. Do not require Spark, Kafka, Flink, cloud accounts, Kubernetes, or containers to begin. The first container dependency is S04 on the Spark route and K02 on the Kafka route; F01–F04, S01–S03, K01, and ST01–ST02 run without containers.

Build engine-independent mental models using small deterministic Python experiments before introducing Spark.

Use real Spark plans, task metrics, and controlled datasets to convert foundations into production troubleshooting expertise.

Build explicit reasoning about partition-scoped ordering, replication, producer acknowledgement ambiguity, consumer progress, and replay. K01 teaches the log on a Python simulator so the Kafka route has no container dependency before K02.

Teach temporal and stateful ideas independent of engines, then apply them in an already-familiar Spark model.

Deepen stateful streaming understanding through an explicit dataflow architecture and operational checkpoints.

Make learners reason about system-wide guarantees, design options, and failure windows across independent technologies.

The order is a partial ordering, not one compulsory 24-item playlist. Diagnostic gaps should drive personalized pathways.

Solid arrows are direct required edges; dotted arrows into a hexagon form an OR-slot: any one of them satisfies the slot.

A required edge is satisfied by a module pass, a module diagnostic, or a passed track gate for that module's track (§8). An OR-slot is satisfied when any one listed module is satisfied by any of those means. Recommended edges create a short bridge lesson or reference, never an automatic block. Unknown level ≠ a prerequisite failure; assess first.

1. Test functional invariants before measuring performance.
2. Assess unseen inputs/failures, not memorization of the guided example.
3. Require an observable repair and regression evidence for Stage 3.
4. Require at least two plausible options and explicit constraints for Stage 4.
5. Distinguish engine-internal recovery from external sink effects; do not award credit for unqualified "exactly once" claims. The same rule applies to batch output: a green job with partial or duplicated published files is a failure.
6. Publish transparent rubric criteria; keep hidden test fixtures and full answers separate.
7. A module must teach, or inherit from a prerequisite, the testing technique its Stage 3 regression test requires.
8. Outcomes are worded by what the module's reference runtime can show: demonstrate only where demonstrable, otherwise explain.

Do not require every Junior to complete 19 modules or assume every Staff engineer has mastered all basics. Readineer should recommend the smallest dependency-complete path that resolves diagnosed gaps, choosing the learner's known engine for any OR-slot.

Distributed Data Processing owns computation distribution, partitioning, execution graphs, data movement, shuffles, joins/aggregations, compute state, event-time processing, processing-engine recovery, stream log delivery, runtime performance, batch output commit behavior at the engine layer, testing of distributed computation, and source-to-sink integration reasoning.

It does not own workflow DAG scheduling and business-level backfills (Pipeline Orchestration & Reliability), nor logical data modeling, Parquet/table layout internals, or Iceberg/Delta transaction protocols (Data Modeling & Storage Design). Integrated labs may reference these mechanisms without duplicating their primary instruction.

1. Approve this concept/prerequisite baseline. Maintain the decision log (§11) for any further change to ownership, direct prerequisite edges, OR-slots, or core/advanced status.
2. Update the companion teaching specification for F04's two new owned concepts (idempotency/scope-naming; testing retry-safe computation), then pilot F01–F04 and G-F, including controlled failures and unit tests. Use the same pilot to validate the Stage 4 decision-record format, the structural pre-check, assessor calibration, and the module-diagnostic format.
3. Validate teaching efficacy: Ask pilot learners to predict behavior before code execution, fix previously unseen failures, and transfer to an unfamiliar scenario. Observe where they get stuck and refine content.
4. Build in parallel after the F pilot — these streams do not depend on each other:
   • S01–S06, including the standalone cluster, its failure controls, and the optional object-store sink; then G-S.
   • K01–K04, simulator first, then the three-broker cluster; G-K should not require Spark.
   • ST01–ST02 on local data sources.
5. Build ST03 once S02 and ST02 exist; introduce Kafka only when it teaches a relevant boundary; then G-ST.
6. Build I01–I02 and G-I once K04, S03, and ST03 exist. The Flink alternatives for slots J and R are wired in when FL ships; until then the Spark path satisfies them.
7. FL kickoff: decide the Flink programming language first; then build FL01–FL04 and G-FL; then I03 and G-P, adding the FL alternatives to slots J, R, and FT.
8. Integrate personalization: Route users using observed competencies, prerequisite diagnostics, and the learner's known engine for OR-slots — not job titles or completed-content counts alone.