How Distributed Jobs Recover from Worker Failures

In F03, we learned how distributed systems use multiple processing stages to calculate results from data spread across workers. Now let’s examine what happens when one of those workers fails during execution. Our nightly revenue-processing job has been extended to produce a customer ledger. A ledger records the calculated result for each customer and makes it available for downstream reporting.

The job now has three stages.

Understanding the Three Processing Stages

1. Map Stage
The input data is divided into 12 partitions. Each map task reads its assigned partition and calculates partial customer-level results. Because customer records may be spread across multiple partitions, these results are not necessarily complete.

2. Reduce Stage
The reduce tasks receive intermediate results from the map tasks through a shuffle. They combine partial results belonging to the same customer to calculate the final customer-level values.

3. Publish Stage
The publish tasks write the completed results to the ledger table. In our reference dataset, the job publishes 796 customer records.
The cluster runs:

• 12 map tasks
• 12 reduce tasks
• 12 publish tasks
• 12 workers distributed across 4 nodes

How Do These Stages Depend on Each Other?

The stages cannot run independently. For example, a reduce task needs the intermediate results produced by the map stage. Similarly, a publish task needs the completed result from its corresponding reduce task. These relationships are called task dependencies.

In our simulator:

• Every reduce task depends on the outputs of all 12 map tasks because any map partition may contain records for its assigned customers.
• Each publish task depends on one corresponding reduce task.

The dependencies form an execution graph.

12 Map Tasks –> 12 Reduce Tasks -> 12 Publish Tasks –> Customer Ledger

The arrows represent dependencies. A downstream task cannot complete its work until the required upstream results are available.

Who Coordinates the Execution?

A distributed job needs a process responsible for assigning work and monitoring its progress. This process is called the coordinator.

In our simulator, the coordinator:

1. 01Identifies which tasks are ready to execute.
2. 02Assigns those tasks to available workers.
3. 03Monitors task completion and failures.
4. 04Tracks which intermediate outputs have been produced.
5. 05Schedules retries or re-computation when necessary.

For example, when a map task finishes, the coordinator records that its output is available. When all required map outputs are ready, the coordinator can schedule the reduce tasks.

How Long Does the Job Normally Take?

On a successful run, our simulator reports the following task durations:

The final publish task completes at 118.9 seconds on the simulator’s execution timeline. As we learned in F01, the job also includes startup and coordination overhead.

Therefore: Total runtime = 10 + 40 + 36 + 118.9 = 204.9 seconds

The job completes successfully and publishes the expected 796 customer records. But what happens if a worker fails while processing a task? Will the entire job need to restart, or can the system recover only the work that was lost?

Let’s begin with the simplest failure scenario. A task fails, but the worker running it remains healthy. To understand recovery, we first need to understand a task attempt.

What Is a Task Attempt?

A task attempt is one execution of a task on a worker. For example, suppose Map Task 7 requires 110.5 seconds to process its input. If it completes successfully, the coordinator records the result and marks the task as finished. But suppose it crashes halfway through. The task has already performed some work, but it hasn’t produced a completed output that the coordinator can use. The coordinator must decide whether to retry it. When the task starts again, that execution is called a second attempt.

In our simulator, the retry starts processing the task’s input from the beginning.

Scenario 1: A Task Fails, but Its Worker Survives

Consider Map Task 7:

• Normal execution time: 110.5 seconds
• Failure point: 50% of execution
• Worker: Worker 3, which remains healthy
• Recovery: Retry the task immediately

The task fails after approximately 55.2 seconds.

Why Isn’t the Additional Delay 110 Seconds?

The second attempt does take another 110.5 seconds. However, the original successful execution would also have required those 110.5 seconds. The additional time comes from the 55.2 seconds spent on the failed attempt. This distinction is important when measuring the cost of retries. The job pays for the processing that was completed before the failure but could not be reused.

Scenario 2: What If the Worker Itself Fails?

Now suppose Worker 3 crashes while running Map Task 7. This is different from a task failure because the worker is no longer available to execute another attempt.

In our simulator:

1. 01Worker 3 fails after 55.2 seconds.
2. 02The worker remains unavailable for 30 seconds.
3. 03It becomes available again at 85.2 seconds.
4. 04The coordinator retries Map Task 7 from the beginning.

The retry still requires the full 110.5 seconds, but it cannot start immediately. The job now takes 290.1 seconds, compared with 204.9 seconds on a successful run. That’s an additional 85.2 seconds.

Comparing Task Failure and Worker Failure

The worker failure takes longer because the task loses its progress and must wait before it can restart. In this particular scenario, we haven’t yet considered whether the failed worker was holding completed outputs from earlier tasks. That introduces a more serious problem.

So far, we’ve examined a task that fails halfway through execution. Now let’s consider a more surprising situation. What happens when a worker fails during a short task but also holds the completed output of a much longer task?

Understanding Where Intermediate Results Are Stored

In our simulator, when a map task finishes, its output remains stored on the worker that executed it. For example:

• Map Task 5 runs on Worker 4.
• It completes after 101.6 seconds.
• Its intermediate output remains on Worker 4.
• Reduce tasks can read that output when they need it.

This works as long as Worker 4 remains available. But if Worker 4 fails, its locally stored intermediate results are lost. The important distinction is that completed computation does not necessarily mean its output is safely stored. If an output exists only on a failed worker, the coordinator may need to recreate it.

Scenario: A Worker Fails During a Short Reduce Task

Consider Reduce Task 4. It requires only 2.9 seconds to complete. Worker 4 starts processing it at 110.5 seconds on the execution timeline. Halfway through the task, at approximately 111.9 seconds, Worker 4 fails. At first glance, this seems like a small problem. The reduce task had completed only about 1.4 seconds of work. Surely retrying a three-second task shouldn’t delay the entire job by much.

But there’s a complication. Worker 4 also holds the completed output from Map Task 5. When the worker fails, Map Task 5’s output disappears with it. Reduce Task 4 needs that output to complete its calculation.

Worker 4 fails halfway through Reduce Task 4, which normally takes 2.9 seconds. The worker also holds the completed output of Map Task 5, which required approximately 101.6 seconds to produce. The coordinator can recompute lost intermediate results.

Step 1: Recompute the Lost Map Output

Map Task 5 had already completed successfully. However, its intermediate output was stored only on Worker 4. When Worker 4 failed, that output was no longer available. The coordinator therefore scheduled Map Task 5 to run again on another worker. Recomputing it required approximately 101.6 seconds.

Step 2: Retry the Reduce Task

Once Map Task 5’s output was available again, Reduce Task 4 could restart. It required another 2.9 seconds to complete.

Step 3: Complete the Publish Task

After Reduce Task 4 finished, Publish Task 4 could write the resulting customer records to the ledger. The complete job finished later because these tasks were part of the job’s remaining dependency chain.

How Much Longer Did the Job Take?

The job took 104 seconds longer, even though the worker failed during a task that required only 2.9 seconds. Why? Because the expensive part of recovery wasn’t the reduce task itself. It was recreating the completed map output that had been lost.

How Does the Coordinator Know What to Recompute?

This is where lineage becomes important. Lineage describes how an output was produced and which earlier computations it depends on. For our example, the relevant dependency chain is:

Map Task 5 ───► Reduce Task 4 ────► Publish Task 4 ────► Customer Ledger
           Produces                          Produces                               Writes
        intermediate                        customer                               final results
           output                               results

When Worker 4 fails, the coordinator identifies that Map Task 5’s output is missing. It then uses the dependency information to determine which tasks need to run again. In our example:

1. 01Recompute Map Task 5.
2. 02Retry Reduce Task 4.
3. 03Continue with Publish Task 4.

Other completed map outputs that remain available do not need to be recomputed. This is the key benefit of lineage-based recovery: the system can reconstruct lost results by repeating the computations that produced them, rather than automatically restarting the entire job.

Why Does This Matter?

A distributed job may contain many intermediate results stored across different workers. When one worker fails, it can lose more than the task currently executing. It may also lose outputs produced by earlier tasks, forcing downstream computations to wait while those outputs are recreated.
The recovery cost therefore depends on:

• Which worker failed.
• Which completed outputs were stored on that worker.
• Which downstream tasks depend on those outputs.
• How expensive the lost computations are to repeat.
• Whether another usable copy of the output exists.

In our simulator, intermediate outputs exist only on their producing workers. Real processing systems may preserve some outputs through external storage or other recovery mechanisms. We’ll examine how preserving intermediate results changes recovery behavior in the next section.

In the previous section, we saw how a worker failure during a 2.9-second reduce task increased the job’s runtime by 104 seconds. The problem wasn’t the failed reduce task itself. The worker also held the completed output of Map Task 5, which had taken more than 100 seconds to produce. When that output was lost, the coordinator had to recompute Map Task 5 before retrying the reduce task.
But what if we could preserve intermediate results so they survive worker failures?

Preserving Intermediate Results

In our simulator, completed map outputs are normally stored on the workers that produced them. If a worker fails, its locally stored outputs disappear. One way to prevent this is to persist intermediate results in durable storage, separate from the worker’s local storage.

For example, after each map task finishes:

1. 01The task writes its intermediate output.
2. 02The output is copied to durable storage.
3. 03The coordinator records that a recoverable copy is available.
4. 04If the original worker fails, downstream tasks can use the saved copy without recomputing the map task.

This approach introduces an additional storage cost, but it can significantly reduce recovery time. In our model:

• Total map output: 0.31 GB
• Durable storage write rate: 100 MB/s
• Storage charge: 0.10 units per GB

The transfer charge for saving the map outputs is: 0.31 × 0.10 = 0.031 units, approximately 0.03 units per run.

Now let’s see whether preserving this output improves recovery.

Recall our earlier failure:

• Worker 4 fails halfway through Reduce Task 4.
• Reduce Task 4 normally takes 2.9 seconds.
• Worker 4 also holds Map Task 5’s completed output.
• Recomputing Map Task 5 requires approximately 101.6 seconds.

Why Is Recovery So Much Faster?

Without persisted outputs, the coordinator must execute the following recovery sequence:

1. 01Recompute Map Task 5, taking approximately 101.6 seconds.
2. 02Retry Reduce Task 4, taking another 2.9 seconds.
3. 03Continue with Publish Task 4.

The complete job takes 308.9 seconds. With persisted map outputs, the coordinator can retrieve Map Task 5’s output from durable storage. It no longer needs to repeat the expensive map computation. The recovery sequence becomes:

1. 01Retrieve the saved map output.
2. 02Retry Reduce Task 4 on an available worker.
3. 03Continue with Publish Task 4.

The job now completes in 208.9 seconds, only 4.1 seconds longer than the reference successful run. Persisting intermediate results reduces recovery time by avoiding unnecessary re-computation. However, saving those results also introduces storage overhead on successful runs, even when no worker fails.

What If We Restart the Entire Job Instead?

There’s another possible recovery strategy: restart everything from the beginning. Under this approach, the coordinator discards the progress made so far and executes every stage again.

For the same worker failure, the simulator reports:

• Total runtime: 346.3 seconds
• Additional runtime: 141.4 seconds
• Previously completed work wasted: 1,273 core-seconds

The restart repeats work that had already completed successfully, including tasks whose outputs were not lost. Compare the three approaches.

The difference is significant. A full restart is simple to understand, but it can be expensive because it discards progress that could otherwise be reused.

Lineage-based recomputation avoids repeating unaffected tasks, while durable intermediate outputs can shorten the recovery process further.

We’ve now explored three approaches to recovering a distributed job:

1. 01Recompute lost results: Repeat only the computations needed to regenerate missing outputs.
2. 02Use persisted outputs: Recover from previously saved intermediate results.
3. 03Restart the entire job: Discard the current execution and begin again.

Which strategy should we choose? The answer depends on what work has already been completed, which outputs are still available, and how expensive they are to recreate.

Let’s examine another failure scenario.

What Happens When a Worker Fails During Publishing?

Our nightly ledger job has three stages: Map → Reduce → Publish

Map tasks prepare partial customer records, reduce tasks combine those results, and publish tasks write the final customer values to the ledger. Now suppose Worker 4 fails halfway through Publish Task 4. The worker also holds intermediate outputs from earlier stages:

• Map Task 5’s completed output.
• Reduce Task 4’s completed output.

If these outputs exist only on Worker 4, both may be lost when the worker fails. The coordinator must determine what needs to be recreated before publishing can continue.

Comparing Recovery with Different Persisted Outputs

Our reference simulator produces the following results:

Let’s understand why these configurations behave differently.

Strategy 1: Nothing Is Persisted

When Worker 4 fails, both its map and reduce outputs are lost. The coordinator must recompute Map Task 5, retry Reduce Task 4, and then retry Publish Task 4. The job takes 314.0 seconds, approximately 109.1 seconds longer than the reference successful run.

Strategy 2: Persist Map Outputs

Now suppose the completed map outputs have been saved to durable storage. When Worker 4 fails, Map Task 5’s output remains available. However, Reduce Task 4’s output is lost because it wasn’t persisted.

The coordinator needs to:

1. 01Reuse the saved map output.
2. 02Recompute Reduce Task 4.
3. 03Retry Publish Task 4.

The job completes in 210.9 seconds, approximately 6 seconds longer than the reference run.

Strategy 3: Persist Reduce Outputs

Finally, suppose the reduce-stage outputs are saved to durable storage. When Worker 4 fails, its local map output may be lost, but Publish Task 4 doesn’t need that map output directly. It only needs the completed result produced by Reduce Task 4. Since the reduce result is safely stored, the coordinator can retry Publish Task 4 without recomputing either earlier task. The job completes in 207.8 seconds, approximately 2.9 seconds longer than the reference run.

Which Stage Should We Persist?

The best choice depends on where failures occur and how expensive recovery would be. For example:

• Map outputs: About 0.31 GB. Useful for avoiding expensive map recomputation when reduce tasks fail.
• Reduce outputs: About 0.04 GB. Useful for restarting publish tasks without repeating earlier computation.

Persisting both stages requires approximately 0.35 GB of durable storage per run in our model.
However, we shouldn’t automatically persist every intermediate result. Saving data requires additional storage, I/O, and potentially network transfers. We need to balance: The cost of preserving intermediate results against the cost of recomputing them after failures.

A frequently failing or computationally expensive stage may justify persistence, while a cheap, easily recomputed stage may not.

What Happens When Retries Keep Failing?

Retries are useful for temporary failures. For example, a worker may become unavailable, or a task may fail because of a transient infrastructure problem. But what if the task fails every time because its code contains a bug? Repeated retries won’t fix the underlying problem. To prevent unlimited retries, distributed processing systems use a retry budget
Our simulator uses: max_attempts = 4
This means a task can execute up to four times, including its initial attempt. Suppose Map Task 7 repeatedly fails halfway through execution. The simulator records these attempts:

After the fourth failed attempt, at approximately 221 seconds, the coordinator aborts the job. In this simulator, the aborted job leaves the ledger table empty. The important lesson is that retries help recover from temporary failures, but persistent failures require investigation and correction.

One More Important Consideration: External Writes

So far, we’ve focused on recomputing intermediate results. But our job also writes to a ledger table. What if a publish task fails after writing some records but before reporting success? Retrying the task could write those records again. Saving intermediate results doesn’t automatically solve this problem. To recover safely, the system also needs a strategy for handling repeated external writes, such as idempotent operations or transactional publishing.

We’ll explore these guarantees in F04.2.

Our experiments demonstrate three important lessons about failure recovery in distributed processing.

1. 01Failed tasks may need to restart from the beginning: When a task fails before producing a reusable result, the coordinator may need to repeat its computation.
2. 02Worker failures can trigger expensive recomputation: A worker may hold completed intermediate outputs from earlier tasks. If those outputs are lost, downstream recovery can require repeating much more work than the task that failed.
3. 03Persisting intermediate results can reduce recovery time: Saving outputs to durable storage allows the coordinator to reuse them after worker failures, but persistence introduces additional costs even on successful runs.

Limitations of the Model

Our simulator makes several simplifying assumptions:

• One failure at a time: Each scenario injects a single task or worker failure. Real distributed systems can experience multiple failures simultaneously.
• Coordinator remains available: We assume the coordinator never fails. In production, coordinator or driver failures may require different recovery mechanisms.
• Durable storage remains available: Once intermediate outputs are persisted, the simulator assumes those copies survive worker failures and remain readable.
• Simplified failure detection: Worker recovery delays and failure-detection timings are predefined. Real systems may take longer to detect failures or allocate replacement resources.
• Reproducible computations: The initial examples assume tasks can be safely recomputed using the same input and configuration. We’ll examine why changing inputs or time-dependent behaviour can break this assumption in the next section.
• Simplified publishing: The simulator models the ledger using controlled write rules. It doesn’t demonstrate every failure that can occur when writing to an external database or storage system.

What This Means for Production Systems

Real distributed processing engines use different recovery mechanisms depending on how tasks execute and where intermediate data is stored. For example, Spark can use lineage to recompute lost partitions. However, cached data and durable checkpoints are not necessarily equivalent. Keeping a result in executor memory or on a worker’s local disk does not guarantee it will survive that worker’s failure. Similarly, successfully recovering a task doesn’t automatically guarantee that an external sink contains correct results.

We’ll study these differences in more detail in the Spark fault-tolerance module, S06.

So far, we’ve assumed that when a task fails, the coordinator can retry it and produce the same result. But that assumption doesn’t always hold. What happens if the data available to a task changes between its original execution and its retry?

Let’s explore why deterministic processing is important for reliable recovery.

What Is Deterministic Processing?

A computation is deterministic when it produces the same output for the same input and configuration, regardless of when or where it runs. For example, suppose a task calculates the total revenue from a fixed set of orders. If the input records and calculation rules remain unchanged, the task should produce the same result every time. This is important in distributed systems because tasks may need to run again after failures. If a retry produces different results, the job may become inconsistent even when all tasks eventually complete successfully.

Example: Processing Late-Arriving Order Events

Our nightly ledger job processes order events within a defined reporting window. However, some events arrive after that window closes. In our reference dataset:

• 217 delivery events arrive after the reporting window closes.
• 51 of those events belong to orders considered by the ledger calculation.
• 26 of those late order events belong to Map Task 7’s input partition.

The job starts two minutes after the reporting window closes. The important question is:
Should a task process records based on the reporting window or based on which records have arrived when the task runs?

Let’s compare two approaches.

Approach 1: Use a Fixed Reporting Cutoff

The job receives a predefined cutoff timestamp representing the end of its reporting window. Every task uses that same cutoff to decide which records qualify. For example:

if record["ingest_time_utc"] <= cutoff_time:
    process(record)

Here, cutoff_time is provided as a fixed parameter for the run. Even if a task fails and retries several minutes later, it uses the same cutoff. Under this contract, the reference job consistently produces:

• Total revenue: 74,558,550 cents
• Customers: 796

The output remains unchanged across retries because the job uses the same logical input and fixed selection rules. This assumes the source makes the same qualifying records available on each attempt.

Approach 2: Use the Current Time

Now suppose the developer changes the logic to process all records that have arrived by the time the task executes. For example:

from datetime import datetime, timezone

current_time = datetime.now(timezone.utc)

if record["ingest_time_utc"] <= current_time:
    process(record)

The task now uses the current clock time instead of the reporting window’s fixed cutoff. This introduces a problem. If the task retries later, additional late-arriving events may have become available. The second attempt can therefore process a different set of records from the first attempt.

What Happens in Our Dataset?

The job starts two minutes after the reporting window closes. By that time, 29 late order events have already arrived. The clock-based version includes these records, producing:

• Total revenue: 75,090,421 cents
• Customers: 797

This differs from the fixed-cutoff result of 74,558,550 cents. Now imagine Map Task 7 fails and must restart. Its second attempt begins later, when additional events have arrived. Will the job still produce the same result?

Our nightly ledger job selects records based on the current time rather than a fixed reporting cutoff. We run the job twice:

1. 01Normal run: Every task completes successfully.
2. 02Failure run: Worker 3 fails while processing Map Task 7, causing the task to retry later.

Why Are the Results Different?

1. Normal execution
The job starts two minutes after the window closes. Because it uses the current time, it includes late order events that have arrived during those two minutes. The total becomes 75,090,421 cents.

2. Execution with a worker failure
Map Task 7 fails and is retried approximately 85 seconds later. By the time the retry executes, another late order event worth 26,998 cents has arrived. The retry includes that event, increasing the final revenue to: 75,090,421 + 26,998 = 75,117,419 cents

3. Manual rerun
The team runs the same job again seven minutes after the reporting window closes. More late events are now available, so the job produces 75,371,869 cents. The same reporting period now has three different revenue totals.

How Do We Make Processing Deterministic?

The solution is to define the input selection rules independently of the time at which individual tasks execute. For example:

1. 01Define a reporting cutoff before the job begins.
2. 02Pass that cutoff to every task.
3. 03Ensure retries use the same cutoff and logical input.
4. 04Process later arrivals through an explicit correction or reprocessing mechanism rather than silently changing the current run.

In our simulator, using the fixed cutoff produces the same result across repeated executions. We can verify this using an output digest, a compact fingerprint calculated from the final result. When the output is identical, its digest is also identical. In the simulator’s six-run comparison:

• Fixed-cutoff processing: All six runs produce one identical digest.
• Clock-based processing: The six runs produce five different digests.

This demonstrates how a time-dependent input selection rule can make distributed recovery unpredictable.

Why Does This Matter?

Distributed systems routinely repeat tasks because of worker failures, timeouts, or other interruptions. For those retries to reproduce the same calculation, the computation must use consistent inputs and rules. Other sources of nondeterministic behaviour can include:

• Generating new random values during each attempt.
• Reading external tables that change between attempts.
• Using the current time to decide which records qualify.
• Depending on execution order when that order isn’t guaranteed.

A deterministic computation makes it possible to recover lost intermediate results consistently. However, deterministic computation alone does not guarantee that external writes are safe to repeat. A task can calculate the same result twice and still insert it into a database twice.

We’ll explore that distinction in F04.2.

How can we tell when task failures are causing unexpected recomputation or inconsistent results? In our simulator, look for four warning signs:

1. 01Multiple task attempts: A task executes more than once because an earlier attempt failed.
2. 02Previously completed tasks run again: The coordinator recomputes earlier tasks because their intermediate outputs were lost.
3. 03High wasted processing time: Failed attempts and repeated computations consume additional core-seconds without producing new useful results.
4. 04Different results after retries: A task or job produces different outputs even though it is supposed to process the same reporting period.

What Does This Look Like in Production?

Similar symptoms may appear as:

• Task logs showing multiple attempts for the same task.
• Earlier processing stages running again after an executor or worker failure.
• A job taking much longer than expected because completed work must be recomputed.
• Logs reporting repeated failures, such as Attempt 4 of 4.
• Revenue totals or record counts changing after a retry or manual rerun.

For example, our worker-failure scenario involved a reduce task that normally required only 2.9 seconds. However, the failed worker also held a completed map output. The system spent over 100 additional seconds recovering because it first had to recreate that lost output.

What Should You Investigate?

When recovery takes longer than expected, examine:

• Failed tasks: Which task or worker failed, and at what point?
• Lost intermediate outputs: What completed results were stored on the failed worker?
• Dependencies: Which downstream tasks require those missing results?
• Retry behaviour: How many attempts were made, and where did each attempt restart?
• Input consistency: Did the retry process the same logical records as the original attempt?

These measurements help distinguish a simple task retry from a larger recovery operation involving earlier processing stages.

When investigating a task or worker failure, consider five important questions:

1. 01What did the failed task depend on? Identify the upstream outputs it needs and the downstream tasks waiting for its result.
2. 02Which outputs are still available? Determine whether the required intermediate data exists on healthy workers or in durable storage.
3. 03How much work must run again? Check whether recovery requires retrying one task, recomputing its dependencies, or restarting the entire job.
4. 04Will retrying produce the same result? Verify that the task uses consistent inputs and configuration rather than changing values such as the current time.
5. 05Why did the task fail, and can it succeed on another attempt? Distinguish temporary failures from persistent problems that repeated retries won’t resolve.

Also consider whether the task performs external writes. Repeating a deterministic calculation can be safe, but repeating its side effects may require additional safeguards.

Let’s revisit Map Task 7. It normally takes 110.5 seconds to complete. Now suppose Worker 3 fails after the task has completed 50% of its work. Previously, our coordinator could retry the task. But what happens if we set: max_attempts = 1

Remember that the retry budget includes the initial attempt. This means the task is allowed to execute only once.

Why Does the Retry Cost Matter?

Retries allow distributed jobs to recover from temporary problems, such as worker failures. However, unlimited retries could waste resources when the underlying problem is persistent.
A retry cost provides a limit.

• Too few attempts: A temporary failure may cause the entire job to fail unnecessarily.
• Too many attempts: A persistent defect may repeatedly consume resources without making progress.

The correct configuration depends on the workload, failure characteristics, and recovery requirements.

Try It Yourself

When the F04 workbench is available, run a worker-failure scenario:

This example uses 16 workers, rather than the 12-worker configuration used in the earlier examples. The simulator injects a worker failure halfway through a reduce task. The execution timeline should show that Worker 3’s failure also caused the loss of an earlier map output:

outputs lost with worker 3: ['map[8]']

The reference results are:

Without persisted map outputs, the failure adds approximately 79.8 seconds to the job. When map outputs are preserved, recovery requires much less re-computation. Compare the execution timelines to understand which tasks run again and why.

Note: The F04 workbench is planned. These commands are intended for use once it becomes available.

Distributed jobs depend on multiple tasks executing successfully and making their results available to downstream tasks. When a task or worker fails, recovery depends on the task’s progress, the intermediate outputs that remain available, and the dependencies between tasks.

In this lesson, we learned that:

1. 01Task retries may repeat work: When a task fails before producing a reusable output, its computation may need to restart from the beginning.
2. 02Worker failures can lose completed results: Intermediate outputs stored only on a failed worker may need to be recomputed.
3. 03Lineage supports selective recovery: The coordinator can identify which missing outputs must be recreated instead of automatically restarting the entire job.
4. 04Persisting intermediate results reduces recomputation: Durable copies can shorten recovery time, but they introduce additional storage and I/O costs.
5. 05Deterministic processing supports reliable retries: Using consistent logical inputs and parameters helps tasks reproduce the same results after failure.

Our Nightly Ledger Example

Consider the different failures we explored.

The most important observation is that a failure during a short task can cause a much longer recovery when the failed worker also holds expensive intermediate results.