When a Task Writes the Same Data Twice

In F04.1, we saw how a coordinator retries failed tasks. For example, if a map task crashes before completing, the coordinator can execute it again to regenerate its result. This works well for computations that don’t directly modify external systems. However, not every task simply calculates an intermediate result. Some tasks also write data to systems outside the processing job.

Let’s understand why this creates a new problem.

The Difference Between Computing Data and Publishing Data

Our nightly ledger job has three stages:

Map Tasks ─────► Reduce Tasks ─────► Publish Tasks ─────► Customer Ledger

Map stage
The map tasks read delivery records and calculate partial customer-level revenue.

Reduce stage
The reduce tasks combine those partial results to produce final revenue totals for each customer.

Publish stage
The publish tasks write the completed customer records to a ledger table used by the finance team. The important distinction is what happens when these tasks are repeated.

Scenario 1: Retrying a Calculation

Suppose Reduce Task 2 successfully calculates its customer-level totals. However, the coordinator doesn’t receive its completion message. The coordinator assumes the attempt failed and runs the reduce task again. If the computation is deterministic, the second attempt produces the same result.

In our simulator, the coordinator manages the reduce output, so the repeated computation does not create additional customer rows in the external ledger. The final ledger remains correct. The retry adds processing time, increasing total runtime from 204.9 to 217.2 seconds, but it doesn’t change the financial results. This is an example of a side-effect-free computation. The task calculates a result without independently modifying an external system.

Scenario 2: Retrying a Database Write

Now consider a publish task. The task calculates nothing new. Its responsibility is to write completed customer results to the ledger table. Suppose it writes 50 customer records successfully. Immediately afterwards, the completion message sent to the coordinator is lost.

From the database’s perspective, the operation succeeded. But from the coordinator’s perspective, the task may still be incomplete. The coordinator eventually retries it. The second attempt writes the same 50 records again. What happens next depends on how the ledger table handles repeated writes. If every write inserts a new row, the ledger now contains 100 rows instead of 50 for those customers. If the ledger replaces existing rows using a stable customer key, the final data may remain unchanged. The retry itself isn’t necessarily the problem.

The problem is that the processing system and the external database may disagree about whether the first attempt succeeded.

What Is a Side Effect?

An operation has a side effect when it modifies something outside the computation that cannot simply be discarded when the task restarts. Examples include:

• Inserting records into a database.
• Publishing messages to Kafka.
• Sending emails to customers.
• Calling an external payment service.
• Writing files to an externally visible location.

These operations may have already completed when a task fails or loses communication with its coordinator. Repeating the task can therefore repeat the side effects.

Why Does At-Least-Once Execution Matter?

Distributed processing systems commonly use retries to recover from failures. If the coordinator cannot confirm that a task completed, it may execute the task again. This gives us an important distinction: A task being scheduled again doesn’t mean its previous attempt had no effect.

The earlier attempt may have completed its database writes before losing communication with the coordinator. With at-least-once execution, the same logical work may run more than once. For a deterministic calculation, repeated execution can produce the same result. For an external write, repeated execution can create duplicates unless the receiving system is designed to handle them.

Who Is Responsible for Preventing Duplicate Writes?

The coordinator decides whether a task needs to run again. But the external sink decides what happens when a repeated write arrives. For example:

• An append-only table may insert another row.
• An upsert table may replace an existing record.
• A sink using idempotency keys may recognise and ignore the repeated request.
• A transactional sink may reject a second commit of the same logical task.

These choices determine whether a retry changes the final data.

Let’s examine a real failure scenario from our simulator. Our nightly ledger job calculates customer revenue and publishes the results to a table used for financial reporting. Under normal conditions, the ledger contains:

• 796 customer rows
• 74,558,550 cents in total revenue

Each customer should have one revenue record for the reporting period.

What Happens to Publish Task 3?

Publish Task 3 is responsible for 78 customers, representing 7,139,987 cents in revenue
The task starts on Worker 3 at 113.1 seconds. It writes all 78 customer records successfully and finishes at 118.9 seconds. However, something unexpected happens. The task’s completion message never reaches the coordinator. The coordinator therefore has no confirmation that Publish Task 3 finished successfully. After waiting for the configured timeout of 10 seconds, it declares the attempt failed and schedules another attempt.
Let’s examine the sequence.

Notice that the first attempt never failed to write its data. The failure occurred while reporting its completion. From the coordinator’s perspective, retrying was reasonable because the outcome was unknown. But the ledger table had already received the first set of writes.

What Happens When the Sink Uses Append-Only Writes?

Our initial ledger implementation uses a simple append operation. Every insert adds a new row, even if the customer already has a record. For example, suppose the first attempt writes:

When the second attempt runs, it writes those records again. The ledger now contains:

No additional orders have been placed. No new revenue has been generated. But the ledger now contains duplicate customer records. If the dashboard sums all rows by customer, the revenue totals will be incorrect.

Why Is This Failure Dangerous?

The original source records were correct. The aggregation logic was correct. The publish task also calculated the correct customer-level values. The error appeared because the same logical results were written to an external system twice. This makes the problem harder to identify.

An engineer investigating the issue might initially search for duplicate source events or incorrect aggregation logic. But neither caused the overstatement. The actual cause was: The first publish attempt succeeded, its completion message was lost, and the coordinator repeated its external writes.

Did the Retry Increase Job Runtime?

Yes. The reference job normally completes in 204.9 seconds. With the lost completion message and retry, it takes 220.7 seconds. That’s an additional 15.8 seconds. However, the more serious consequence isn’t the additional runtime. It’s that the job reports successful completion while the ledger contains incorrect revenue.

Could We Prevent This Without Disabling Retries?

We could try disabling retries for publish tasks. But that would create another problem. If a task genuinely fails before completing its writes, the ledger could be left incomplete. The coordinator needs a way to recover from uncertain outcomes without creating duplicate effects. The better approach is to make repeated writes safe. This is where idempotency becomes important.

We’ve seen how an append-only ledger produces duplicate records when a publish task runs more than once. Now let’s keep the same failure but change how the ledger handles writes. Our goal is straightforward: the coordinator should be able to retry Publish Task 3 without changing the correct final ledger result. This property is called idempotency.

What Does Idempotency Mean?

An operation is idempotent when repeating the same logical operation has the same final effect as executing it once. For example, suppose a customer has a revenue total of 5,000 cents. An append operation might behave like this:

Attempt 1: INSERT (cust-001, 5000)
Attempt 2: INSERT (cust-001, 5000)

Final result:
cust-001 → 5000
cust-001 → 5000

The table contains two rows. Now consider an upsert:

Attempt 1: UPSERT (cust-001, 5000)
Attempt 2: UPSERT (cust-001, 5000)

Final result:
cust-001 → 5000

The second operation replaces the same customer record with the same value. The final result remains unchanged. This is what we want from a retry-safe sink.

Let’s compare four possible implementations.

Strategy 1: Append-Only Sink

An append-only sink inserts a new row for every write. It doesn’t check whether the same customer has already been written for the current reporting period.
When Publish Task 3 runs twice:

• Attempt 1 writes 78 customer records.
• Attempt 2 writes those same 78 records again.
• The sink retains all 156 writes from the two attempts.

The final ledger contains: 874 rows instead of 796
The revenue is overstated because the dashboard counts duplicate customer records. This sink isn’t idempotent for our current write pattern.

Strategy 2: Upsert Sink

An upsert uses a stable business key to identify an existing record. For example, the ledger could identify each record using: (reporting_period, customer_id)
When a write arrives:

• If the record doesn’t exist, insert it.
• If the record already exists, update its value.

Now suppose Publish Task 3 runs twice. The first attempt inserts or updates its 78 customers. The second attempt performs an upsert using the same keys and the same values. Instead of creating duplicate rows, it overwrites the same records. The ledger still contains 796 rows with the correct revenue totals.

Why Does This Work?

The retry represents the same logical computation. The customer revenue values are deterministic and don’t change between attempts. Therefore, replacing an existing value with the same value doesn’t change the final result. However, there’s an important limitation.
An upsert is only retry-safe for this scenario because repeated attempts write the same intended values using the same stable keys.

If two attempts calculate different values, or an older attempt overwrites a newer legitimate update, a simple last-write-wins upsert may not provide the required correctness guarantee. For our current fixed-input nightly ledger, it works.

Strategy 3: Use an Idempotency Key

Instead of overwriting existing records, another approach is to recognise repeated operations and ignore them. We can assign each logical write a unique, stable identifier called an idempotency key. For our simulator, the identifier contains: (job, task, customer)

For example: (nightly-ledger-2026-03-02, publish-3, cust-0008)

The sink maintains a record of keys that have already been processed. When a new write arrives:

1. 01Check whether its idempotency key has been seen before.
2. 02If not, apply the write and record the key.
3. 03If the key already exists, ignore the repeated write.

The sink must make the key check and its associated data write atomic so concurrent attempts cannot both pass the check.

What Happens During the Retry?

Publish Task 3 writes its 78 customers during Attempt 1. Each write has an idempotency key identifying the logical job, task, and customer. When Attempt 2 runs, it submits the same 78 records with the same keys. The sink recognises that all 78 writes have already been processed. It ignores them. The final ledger remains: 796 customer records with the correct values.

Notice that the key doesn’t need to identify which attempt performed the write. It needs to identify the logical operation we’re trying to complete. We’ll explore why this distinction matters in the next section.

Strategy 4: Use an Atomic Commit per Task

Our fourth strategy is to make an entire publish task’s output visible through a single atomic commit. Instead of immediately exposing each customer row, the task first prepares its results in a staging area. Once all rows are ready, the task requests a commit. The sink then makes the task’s complete output visible together.

For example:

Publish Task 3 –> Stage 78 customer records -> Commit logical task: publish-3 –>
78 records become visible

The sink also records that the logical task has already committed successfully. If the coordinator retries the same task, the retry may prepare the rows again, but its second commit is rejected or treated as already completed. The original committed rows remain visible. This prevents a second attempt from publishing the same logical task output again.

It also provides an additional benefit: if the task fails before committing, its partially staged rows don’t become visible as a completed result.

Important: This requires a commit protocol that both makes the writes atomic and recognises that the logical task has already committed. A database transaction by itself does not necessarily prevent two successful transactions from inserting duplicate records.

Let’s repeat the same failure from Section 02.
Publish Task 3:

1. 01Successfully writes its 78 customer records.
2. 02Loses its completion message.
3. 03Is retried by the coordinator.
4. 04Produces the same customer records during Attempt 2.

Why Is the Runtime the Same for Every Strategy?

All four configurations experience the same coordinator failure. The completion message is still lost. The coordinator still waits for the timeout. Publish Task 3 is still retried. Therefore, the modelled job runtime remains 220.7 seconds in every case. What changes is how the sink handles the second attempt:

• The append sink creates duplicate rows.
• The upsert replaces previously written values.
• The idempotency-key sink recognises and ignores repeated operations.
• The atomic sink prevents a second commit of the same task output.

In this simulation, each strategy has the same task timing. Real implementations may have different costs for index lookups, constraint checks, staging, and transactions.

Which Strategy Should We Choose?

Each approach offers different behaviour.

For our nightly ledger, all three retry-safe approaches produce the expected 796 customer records. But correctness depends on more than choosing a sink type. It also depends on how we identify the logical work being repeated. For example, an idempotency key that changes on every retry won’t identify the retry as a duplicate. The sink would treat it as a new operation. That’s the next problem we’ll investigate.

In the previous section, we learned how idempotent sinks can prevent duplicate writes when a task executes more than once. Our nightly ledger job can now retry Publish Task 3 without corrupting the final results. The team decides to use an idempotency key to identify repeated writes. However, there’s a subtle problem with how they design the key.

The Idempotency Key That Looks Correct

Recall that our coordinator may execute the same task multiple times. For example:

• Attempt 1: Publish Task 3 writes 78 customer records, but its completion message is lost.
• Attempt 2: The coordinator retries Publish Task 3, producing the same 78 records.

The sink maintains a record of previously processed idempotency keys. If it receives a write with an existing key, it ignores that write. To ensure every write has a unique identifier, the developer chooses: (job, task, attempt, customer)

For example, the first attempt writes: (nightly-ledger, publish-3, attempt-1, cust-0008)
The second attempt writes: (nightly-ledger, publish-3, attempt-2, cust-0008)

At first glance, this seems reasonable. Each write has an identifier, and no two attempts share the same identifier. But that’s exactly the problem. The idempotency key identifies the execution attempt, not the logical operation being repeated. When the coordinator retries the task, the attempt number changes. The sink sees a new key and assumes it’s a new operation, even though the customer record was already written successfully.

Why Does Removing the Attempt ID Fix the Problem?

Consider the purpose of an idempotency key. We don’t want every request to have a different identifier. We want multiple attempts to perform the same logical operation using the same identifier. For example, the logical operation might be: Publish the revenue calculated for customer cust-0008 by Publish Task 3 in this nightly ledger run.

That operation should have the same identity regardless of whether it executes once or five times. Therefore, a better key is: (job, task, customer)

Here, job identifies the specific logical ledger run or reporting period, not just a reusable job name. Both attempts now submit:

Attempt 1:
(nightly-ledger-2026-03-02, publish-3, cust-0008)

Attempt 2:
(nightly-ledger-2026-03-02, publish-3, cust-0008)

The sink recognises the second write as a duplicate and ignores it. The final ledger remains correct.

Why Could This Bug Escape Testing?

The implementation might pass a simple duplicate-write test. For example, a developer could submit the same record twice using the same attempt ID. The sink would correctly ignore the second write. However, that test doesn’t reproduce what actually happens during task recovery. When the coordinator retries a task, it creates a new attempt ID. The same logical records are then written using different idempotency keys. The original test passes, but the real retry still creates duplicates. A stronger test must simulate the coordinator retrying the same logical task with a different attempt ID.

What Is the General Rule?

An idempotency key should identify the logical operation whose effect must occur once, not the particular execution attempt. For our ledger, that might include the reporting period, task identity, and customer identity. The exact key depends on the sink’s business contract. For example, a table storing one balance per customer and reporting period may use (reporting_period, customer_id) as its logical write identity. What matters is that retries of the same operation reuse the same key, while genuinely new operations receive different keys.

So far, we’ve examined a relatively straightforward failure. Publish Task 3 completed all its writes, but the coordinator never received its completion message. The task ran again, creating duplicates in an append-only sink. However, duplicate writes can occur in several other situations. Let’s examine three common failure scenarios.

Scenario 1: A Worker Fails After Writing Some Records

Suppose Publish Task 4 is responsible for 73 customers. It begins writing customer revenue records to the ledger. However, the worker crashes after completing approximately half its work. Before the failure, the task has already written 36 customer records. The coordinator detects the failure and retries Publish Task 4 on another worker. The second attempt starts from the beginning and writes all 73 customer records. What happens to the 36 records written before the failure? That depends on the sink. An append-only table doesn’t automatically remove rows just because the task that wrote them has failed. Those 36 rows remain in the database. When the retry writes all 73 rows, the first 36 are inserted again.

Why Does an Atomic Commit Behave Differently?

With append-only writes, each customer record becomes visible as soon as it is successfully inserted. If a task fails after writing 36 records, those records remain visible. An atomic task commit works differently. The task first stages all its records. Only after the complete output is ready does it commit the batch. The sink makes the entire set visible together.

For example:

Attempt 1: Publish Task 4
        |
        v
Stage 36 records
        |
        v
Worker fails
        |
        v
No commit -> records not visible


Attempt 2: Publish Task 4
        |
        v
Stage all 73 records
        |
        v
Atomic commit
        |
        v
73 records become visible

The first attempt never committed its output. Therefore, its 36 staged records don’t appear in the final ledger. The second attempt successfully commits all 73 records. This prevents both duplicate rows and partially published task results.

However, remember that atomicity alone isn’t enough to prevent duplicate commits. The sink must also recognise the logical task so that two successful attempts cannot both publish their results independently.

Scenario 2: A Zombie Task Continues Running

Now consider a more unusual failure. Suppose Publish Task 5 is responsible for 59 customers. Its worker loses communication with the coordinator after completing approximately 40% of the task. The coordinator assumes the attempt has stopped making progress. However, the worker is still running. It can also still communicate with the external ledger database. This creates a zombie task: an attempt that the coordinator considers failed even though it continues executing.

What Happens Next?

Our simulator records the following sequence:

1. 01Publish Task 5 loses communication with the coordinator.
2. 02The original attempt continues processing and writes all 59 customer records.
3. 03Attempt 1 finishes at 116.9 seconds.
4. 04The coordinator receives no completion confirmation and declares the attempt failed at 124.2 seconds.
5. 05The coordinator starts Attempt 2.
6. 06Attempt 2 writes the same 59 customer records and finishes at 128.8 seconds.

Both attempts write successfully to the ledger. The coordinator may recognise only one of them as successful, but the database receives both sets of writes.
With an append-only sink: Final rows = 796 + 59 = 855
The duplicate records overstate revenue by 3,844,716 cents.

Why Is This Different from a Normal Retry?

In the earlier examples, the first attempt had either finished or crashed. Here, the first attempt continues running even after the coordinator decides to retry it. The two attempts can therefore overlap. This is why coordinator-side task status alone cannot prevent duplicate external writes. A task that the coordinator considers failed may still perform side effects. To handle this safely, the sink must recognise repeated logical operations or enforce a commit protocol that prevents competing attempts from publishing the same output.

Scenario 3: Speculative Execution Creates Two Attempts

Distributed systems may also launch duplicate task attempts deliberately. This technique is called speculative execution. Suppose the coordinator notices that one task is taking unusually long. Rather than waiting indefinitely, it may start another attempt of the same task on a different worker. The goal is to let the faster attempt finish first. This can improve performance when a task is slow because of temporary worker-level problems. But what happens if both attempts perform external writes?

Our Speculative Execution Example

Publish Task 7 is identified as slow. The coordinator schedules another attempt at 115.0 seconds. The second attempt starts on Worker 11 at 116.3 seconds. The two attempts finish as follows:

Attempt 1 finishes first and is accepted by the coordinator.
Attempt 2 finishes later, so its computational result is ignored.

However, both attempts have already written their customer records to the append-only sink. The ledger ends up containing 867 rows instead of 796. The additional rows overstate revenue by 14,109,253 cents.

Why Didn’t Ignoring the Second Attempt Prevent Duplicates?

Because ignoring an attempt’s result isn’t the same as undoing its external writes. The coordinator can discard an intermediate result that it controls. But once a task inserts records into an external database, those writes may already be visible. Even if the coordinator decides that another attempt won, the losing attempt’s side effects can remain. This illustrates an important risk:
Duplicate writes don’t always require a failure. They can also result from running multiple attempts intentionally.

Comparing the Four Failure Scenarios

We’ve now examined four ways the same logical task can write its results more than once.

All four scenarios reveal the same fundamental issue: Task execution and external side effects are separate concerns. The coordinator can decide which attempt is successful, but it cannot automatically undo writes that have already reached another system.

How Do We Protect Against These Scenarios?

For our nightly ledger, the simulator demonstrates three approaches that maintain the intended results:
1. Upsert using a stable business key
Every attempt writes the same deterministic customer totals using the same record identity. Repeated writes replace existing records rather than create duplicates.

2. Use stable idempotency keys
Every logical write carries a key that remains unchanged across retries. The sink atomically checks whether that operation has already been applied and ignores duplicates.

3. Use an atomic commit protocol
Each attempt stages its output, but only one attempt can successfully commit the logical task’s results. Other attempts are rejected or treated as already completed. For our current workload, a task-level commit offers an additional benefit: incomplete attempts don’t expose partially written customer records. However, each approach has different implementation requirements and failure guarantees.

What About External Systems Other Than Databases?

Our examples use a ledger table, but real distributed jobs may also publish Kafka messages, call APIs, or send notifications. An upsert in a database doesn’t automatically make those other side effects safe. Each external system needs its own retry and idempotency contract.

We’ll examine this limitation in the next section.

Our experiments demonstrate three important lessons about retries and external writes.

1. At-Least-Once Execution Can Repeat Side Effects

When a coordinator cannot confirm that a task completed, it may retry the task. However, the original attempt may already have performed external writes. An append-only sink records those writes again, creating duplicates. In our example, losing the completion message for Publish Task 3 caused 78 customer records to be written twice. The ledger contained 874 rows instead of 796, and the dashboard overstated revenue by approximately 9.6%.

2. Idempotent Sinks Make Retries Safer

An upsert, stable idempotency key, or properly coordinated atomic commit can prevent repeated attempts from corrupting the final result. However, their guarantees depend on their implementation.
For example:

• An upsert requires appropriate record keys and consistent update semantics.
• An idempotency key must remain stable across retries of the same logical operation.
• An atomic commit protocol must ensure that only one attempt can publish a logical task’s output.

Simply adding an idempotency-key column or using a database transaction doesn’t automatically solve duplicate-write problems.

3. Idempotency Keys Must Identify Logical Work

An idempotency key such as (job, task, attempt, customer) doesn’t prevent duplicates across retries because every attempt has a different identifier. For our simulator, the appropriate identity is: (job, task, customer)
Here, the job identifies the specific logical reporting run, and the task and customer identify the write being performed. The general principle is that repeated attempts of the same logical operation must use the same idempotency key.

Limitations of the Model

Our simulator makes several simplifying assumptions.

• Simplified sink implementations: The ledger is modelled as a table with defined append, upsert, deduplication, and atomic-commit rules. Real databases require appropriate constraints, transactions, and concurrency handling.
• Deterministic task output: We assume retries calculate the same customer values from the same logical input. As discussed in F04.1, changing inputs or time-dependent processing can break this assumption.
• One injected failure scenario at a time: We examine individual failures and overlapping attempts in controlled simulations. Production environments can experience multiple simultaneous failures.
• Healthy coordinator: The coordinator remains available throughout each scenario. Coordinator failures introduce additional recovery and commit-coordination challenges.
• Simplified dashboard behaviour: The dashboard sums all ledger rows for each customer. A dashboard that selects only the latest row might hide duplicates, but the underlying append-only table would still contain them.
• External notification behaviour: The ledger sink’s guarantees apply only to its own writes. Sending an email or calling another service introduces additional side effects that must be handled separately.

What Does This Mean for Production Systems?

A distributed job can process its input correctly, recover successfully from failures, and still produce incorrect external results. For example, Spark may retry tasks after an executor failure or launch speculative attempts when tasks appear slow. If these tasks write directly to an external system, the receiving system must be designed to handle repeated attempts. The appropriate solution depends on the sink. A database may support transactional upserts. A file-based sink may require an output commit protocol. A message broker may provide producer idempotency or transactions with guarantees limited to particular operations.

We’ll examine these mechanisms more deeply in S06: Spark Fault Tolerance, Output Commit & Runtime Debugging and K04: Kafka Delivery Semantics.

The next part of this article will also introduce a critical distinction: even when a ledger table is correct, other side effects, such as sending customer emails, may still happen more than once.

So far, we’ve focused on making our nightly ledger safe when publish tasks execute more than once. We’ve seen how upserts, stable idempotency keys, and atomic commits can prevent duplicate customer records. Our ledger now contains the correct 796 customer records, even when tasks fail or retry. But the reporting team introduces a new requirement. After publishing a customer’s revenue, the system must also send that customer an email notification. This sounds straightforward. Each publish task can write the customer record to the ledger and then call an email service. But what happens if the task executes twice?

A New Side Effect

Our publish task now performs two operations:

1. 01Write the customer’s revenue to the ledger.
2. 02Send an email notification to the customer.

These operations affect two different systems. The ledger database stores the revenue records. The email service delivers messages to customers. Even if both operations happen inside the same publish task, they don’t necessarily share the same transaction or idempotency guarantees. Consider the following workflow:

Publish Task ─────► Ledger Database
      |
      └───────────► Email Service

Suppose the ledger supports atomic commits. This ensures that only one attempt can commit the records belonging to a logical publish task. However, the ledger’s commit rules don’t control what happens inside the email service. A repeated task might fail to commit its ledger records but still send its emails successfully. Let’s examine this using our simulator.

Scenario: A Zombie Task Sends Emails Twice

Recall the zombie-task scenario from Section 05. Publish Task 5 is responsible for 59 customers. Its worker loses communication with the coordinator, but the task continues executing. The coordinator eventually assumes that the task has failed and starts another attempt. Both attempts finish. The ledger uses an atomic commit protocol, so only one attempt can commit the customer records. The other attempt’s commit is rejected. Therefore, the ledger contains the correct 796 rows. But both attempts also call the email service.

What happens to the notifications?

Why Didn’t the Atomic Commit Prevent Duplicate Emails?

The atomic commit protects the ledger database. It ensures that only one attempt can commit a particular logical task’s records. However, the email API is a separate system. When the task sends an email, that message can leave the application immediately. Rejecting a database commit afterward doesn’t undo an email that has already been delivered. This introduces an important idea: A processing guarantee applies only to the operations and systems covered by that guarantee.

For example:

• The ledger may guarantee one committed result per logical task.
• The email service may receive the same notification request multiple times.
• A task may be marked successful only once while its external side effects occur more than once

We must therefore ask not just whether a job provides exactly-once results, but exactly once for which operation and within which system?

The Solution: Use a Transactional Outbox

One way to reduce this problem is to stop sending emails directly from publish tasks. Instead, we can use an outbox pattern. An outbox is a table that stores notifications or messages that need to be delivered later. The publish task writes both the ledger result and the notification request within the same database transaction.

A separate process reads committed notification requests from the outbox and sends them to the email service.

The workflow becomes:

Publish Task ─────► Atomic Ledger + Outbox Commit ─────► Outbox Sender ─────► Email Service

Let’s examine how this works.

Step 1: Write the Customer Result

Suppose Publish Task 5 processes cust-0011. It produces a customer revenue record:

Instead of immediately sending an email, the task prepares a notification record. For example:

The notification ID represents the logical notification, not the task attempt. If the task is retried, it generates the same notification ID.

Step 2: Commit Both Records Together

The publish task commits the customer revenue record and notification record in one atomic transaction. This ensures that the two records become visible together. If the transaction fails, neither record is committed. If the transaction succeeds, both records are available. The task also uses the ledger’s existing idempotency rules so that retries don’t create duplicate logical notifications.

Step 3: Send Notifications Separately

A separate sender reads committed records from the outbox. It calls the email service and records delivery progress. This separates the email API call from the critical path of the nightly ledger computation. The publish task no longer needs to wait for every customer email to be delivered before finishing.

Step 4: Handle Publish-Task Retries

Now suppose Publish Task 5 becomes a zombie and runs twice. Both attempts prepare the same customer records and notification requests. Under our atomic sink’s rules, only one logical task commit succeeds. The second attempt cannot add another set of committed notifications. The outbox therefore contains one logical notification per customer. Our simulator reports:

• 796 ledger rows
• 796 email notifications delivered
• 0 customers emailed twice

The result is correct within the simulator’s modelled failure and delivery rules.

Does an Outbox Guarantee Exactly-Once Email Delivery?

Not by itself. This distinction is important for production systems. A transactional outbox ensures that the database result and the intention to send a notification are recorded together. However, the outbox sender still communicates with an external email API. Consider another failure:

1. 01The sender calls the email API.
2. 02The email is successfully delivered.
3. 03The sender crashes before recording that the notification was sent.
4. 04The sender restarts and processes the same outbox record again.

Without idempotency support from the email service, the customer may receive another email. Therefore, an outbox provides a reliable, transactional record of which notifications need to be sent, but it does not automatically guarantee exactly-once delivery at an external service. A production implementation may need a stable notification ID, receiver-side deduplication where supported, and a clear retry policy.

In our simulator, the outbox sender’s behaviour is simplified so each committed outbox notification is sent once.

How Does the Outbox Affect Performance?

There’s another benefit to separating email delivery from publish tasks. Sending emails directly from the publish task makes the task slower because it must wait for external API operations. In our reference simulator:

• Publish task without direct email sending: Approximately 5.0 seconds.
• Publish task with direct email sending: Approximately 22.8 seconds.

The longer publish tasks increase the overall job runtime:

Moving email delivery to a separate sender allows the ledger job to finish without waiting for all email API calls. The notifications can then be delivered independently. However, this doesn’t eliminate the work or cost of sending emails. It moves that work outside the ledger job’s critical path.

What Happens If the Outbox Uses an Append-Only Sink?

An outbox inherits the correctness properties of the storage system used to write its records. If the underlying sink blindly appends duplicate notification requests, retries can still create duplicate outbox entries. Our simulator demonstrates this:

• Atomic sink with outbox: 796 notifications.
• Upsert sink with outbox: 796 notifications.
• Append-only sink with outbox: 855 notifications.

The outbox must therefore use stable notification identities and appropriate duplicate-prevention rules. Simply adding an outbox table doesn’t make a non-idempotent write process safe.

Understanding the Scope of a Guarantee

We can now distinguish three different guarantees.

Each operation has its own correctness requirements. A guarantee for one operation does not automatically extend to the others.

How can we identify when task retries or overlapping attempts are causing incorrect external results? In our simulator, several measurements help reveal these problems.

Warning Sign 1: More Rows Than Expected

Our ledger is supposed to contain one row per customer. If it contains 874 rows for 796 customers, something is wrong. Possible causes include:

• Repeated append operations.
• Multiple attempts writing the same customer records.
• A missing or incorrect idempotency key.
• A commit protocol that doesn’t prevent duplicate logical writes.

The difference between expected and actual row counts can help identify which task was repeated. For example: 874 − 796 = 78 extra records
Publish Task 3 is responsible for exactly 78 customers. That provides a useful clue.

Warning Sign 2: Revenue Is Higher by One Task’s Contribution

Our correct revenue total is: 74,558,550 cents
After the lost completion message, the dashboard reports: 81,698,537 cents
The difference is: 7,139,987 cents

That is exactly the revenue belonging to Publish Task 3. When an overstatement matches the complete output of one task, investigate whether its results were published multiple times.

Warning Sign 3: Different Attempt IDs for Duplicate Records

Suppose duplicate ledger records have idempotency keys containing different attempt IDs. For example:

(job-01, publish-3, attempt-1, cust-0008)
(job-01, publish-3, attempt-2, cust-0008)

These keys identify different executions of the same logical write. The sink treats them as different operations, even though both attempts wrote the same customer result. This suggests that the idempotency key was designed around the attempt rather than the logical work.

Warning Sign 4: Correct Ledger, Duplicate Notifications

Sometimes the database results are correct, but customers report receiving multiple emails. For example:

• Ledger rows: 796
• Emails sent: 855
• Customers emailed twice: 59

This suggests that the ledger’s idempotency protection works, but notifications are being sent through a separate path that doesn’t share those guarantees.

What Should You Investigate?

When debugging repeated side effects, examine:

• Which task attempts performed external writes.
• Whether the first attempt committed successfully before losing communication.
• Whether multiple attempts ran at the same time.
• Which identifiers the sink uses to recognise duplicate operations.
• Whether the sink commits rows individually or as one logical batch.
• Which external effects fall outside the sink’s transaction.

Before allowing a distributed task to write to an external system, consider these five questions.

1. Does the Task Perform Any Side Effects?

Identify every operation that changes an external system. Examples include database writes, message publishing, API calls, and email delivery. A computation may be safe to retry while its external effects are not.

2. What Identifies the Logical Write?

Determine which fields uniquely identify the operation. For example, a nightly customer ledger might use: (reporting_period, customer_id). A task-level commit might use: (job_run_id, task_id). The identifier must remain stable across retries of the same logical operation. It should not change merely because the coordinator creates a new task attempt.

3. What Happens If a Task Fails After Writing Some Records?

Check whether the already-written rows remain visible. An append-only sink may retain them, leading to duplicates when the task retries. An atomic task commit can prevent partially staged results from becoming visible.

4. What Exactly Does the Sink Guarantee?

Be precise about the scope of correctness.
Does the system guarantee:

• One stored row per customer and reporting period?
• One committed output per logical task?
• No duplicate messages in a particular topic?
• One notification request per customer?

These are different properties. A guarantee covering the ledger doesn’t automatically cover an external email service.

5. What Happens If Two Attempts Execute Together?

Consider zombie tasks and speculative execution. Both attempts may be running, and both may try to perform the same external operation. The receiving system must be able to handle those concurrent requests safely. For example, checking whether an idempotency key exists and recording the new operation must be protected against races between attempts.

Let’s revisit the difference between computational tasks and tasks that perform external writes. Suppose Reduce Task 2 finishes successfully. However, its completion message is lost before reaching the coordinator. The coordinator assumes the attempt failed and runs Reduce Task 2 again. The ledger uses an append-only sink. Will this retry create duplicate customer records?

Try It Yourself

When the F04 workbench is available, compare the sink strategies under a lost-completion-message failure.

Run:

This runs the failure scenario with:

• Five sink strategies.
• Two email-delivery strategies.
• A lost completion acknowledgement for Publish Task 2.
• Sixteen workers and sixteen tasks per map and reduce stage.

The simulator produces 10 configurations for comparison.

What Should You Observe?

In this configuration, Publish Task 2 is responsible for 53 customers. The reference results include:

The append sink creates 53 duplicate customer rows because Publish Task 2 writes its records twice. The upsert sink replaces the 53 existing records without increasing the ledger’s row count. The email results demonstrate a separate issue: direct email sending can produce duplicates even when the database remains correct. The outbox path prevents those duplicate notifications under the simulator’s controlled delivery assumptions.

Questions to Answer After Running the Experiment

Compare the results and explain:

1. 01Why does the append sink create 849 rows?
2. 02Why does the upsert sink retain 796 rows?
3. 03Why does an attempt-based idempotency key fail to identify the retry?
4. 04Why can the ledger be correct while customers receive duplicate emails?
5. 05What additional guarantees would an outbox sender need in a production environment?

Note: The L-F04 workbench is planned. These commands are intended for use once it becomes available.

Distributed systems retry tasks because they cannot always determine whether an earlier attempt completed successfully. However, repeating a task can also repeat its external side effects.

In this lesson, we learned that:

1. 01At-least-once execution can repeat writes. A task may successfully write data but execute again when its completion acknowledgment is lost.
2. 02Append-only sinks can produce duplicates. Retrying a publish task may insert the same records again, causing incorrect counts and totals.
3. 03Idempotent writes make retries safer. Upserts, stable idempotency keys, and properly coordinated atomic commits can preserve the intended result across repeated attempts.
4. 04Idempotency keys must identify logical operations. Including an attempt ID can cause the sink to treat a retry as a new operation.
5. 05Overlapping attempts create additional risks. Zombie tasks and speculative execution can cause multiple attempts to write to an external system, even when the coordinator accepts only one result.
6. 06Correctness guarantees have boundaries. A ledger can contain the right records while an external email service receives duplicate requests.

Our Nightly Ledger Example

The expected result is: 796 customer records and 74,558,550 cents in revenue.

After losing Publish Task 3’s completion message:

The source data and revenue calculations were correct. The incorrect results came from publishing the same customer records twice. Using a retry-safe sink prevents those duplicates. But the zombie-task scenario revealed another problem: 796 ledger rows and 855 emails.

The database was correct because only one logical task commit was accepted. The email service still received duplicate requests because both task attempts called it independently.