A backfill re-runs a daily pipeline over several dates. Each run is a pair (partition, task). Plan the order the scheduler will start them in.
- •
partitions — the dates to backfill ("YYYY-MM-DD" strings, any order). - •
deps — {task: [upstream tasks]} for one partition; every task is a key. - •
max_parallel — the most runs that can start together.
A run can start once:
1. every upstream task has finished for the same partition, and 2. the same task has finished for the previous partition (the next older date in partitions) — each task depends on its own past. The oldest partition has no previous one.
Runs start in waves. A run joins a wave only if everything it waits for finished in an earlier wave. A wave holds at most max_parallel runs; when more are ready, take the oldest partition first, then task name alphabetically. List each wave's runs in that same order.
Write plan_backfill(partitions, deps, max_parallel) returning the list of waves, each a list of (partition, task) tuples. No partitions → [].
Python 3.13 in your browser — the standard library plus pandas and numpy; no pip installs.