Why Data Location Matters in Distributed Processing

Our hourly validation job processes 24 million order events (6.48 GB) using a cluster with:

• 4 machines (nodes)
• 16 workers, with 4 workers on each node
• 16 input splits, distributed across the four machines

Each input split contains a portion of the dataset, and one task is responsible for processing each split. The job completes in approximately 133 seconds, comfortably meeting its deadline. Everything appears to be working well. However, the network team notices something unexpected: approximately 4.8 GB of data moves between the four machines every hour while the job is running. This raises an important question:

If the data is already stored on the cluster, why do we need to transfer so much of it between machines?

Understanding the Problem

When a worker processes data, it must first read the records assigned to its task.
There are two possible situations:

1. 01Local read: The worker processes data stored on the same machine.
2. 02Remote read: The worker processes data stored on another machine, requiring the data to travel over the network.

Remote reads introduce additional network traffic and may increase processing time and infrastructure costs. The relationship between where data is stored and where computation runs is called data locality. An important responsibility of a distributed processing scheduler is deciding which worker should execute a task. If the scheduler considers data location, it may reduce unnecessary network transfers by assigning tasks to workers near their input data.

Why Does This Matter?

In our example, the job completes within its deadline, but nearly 75% of the input data is transferred between machines. Even if these transfers don’t significantly increase runtime, they still consume network bandwidth and may affect other workloads sharing the cluster.

Let’s understand how the scheduler assigns tasks to workers and why some tasks read data locally while others require network transfers.

Step 1: Where Is the Data Stored?

Our dataset is divided into 16 input splits, numbered from 0 to 15. Each split is stored on one of the four cluster nodes. The node holding a split is called its home node.
In our example, the splits are distributed as follows:

Split:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
Node:   2  1  0  1  1  0  2  2  3  3  3  3  0  0  1  2

Each node stores four splits, but the split numbers are not necessarily consecutive.
For example:

• Split 0 is stored on Node 2.
• Split 1 is stored on Node 1.
• Split 2 is stored on Node 0.

The data was placed wherever storage capacity was available during ingestion.

Step 2: Where Are the Workers Running?

Our cluster has 16 workers distributed evenly across 4 nodes. The workers are assigned to nodes using: Worker node = Worker ID % 4

This means:

• Worker 0 runs on Node 0.
• Worker 1 runs on Node 1.
• Worker 2 runs on Node 2.
• Worker 3 runs on Node 3.
• Worker 4 runs on Node 0, and the pattern repeats.

Each worker can process a task independently.

Step 3: How Does the Scheduler Assign Tasks?

Our simulator initially uses a simple scheduling strategy called placement = any.
Under this strategy, the scheduler assigns each task to an available worker without considering where its input data is stored.
Since all 16 workers are initially free:

• Task 0 goes to Worker 0.
• Task 1 goes to Worker 1.
• Task 2 goes to Worker 2.
• The pattern continues until all 16 tasks are assigned.

Now let’s examine what happens.
Example 1: Task 0

• Task 0 needs to process Split 0.
• Split 0 is stored on Node 2.
• Worker 0 runs on Node 0.

Since the data and worker are on different nodes, Worker 0 must read the split over the network. This is a remote read.

Example 2: Task 1

• Task 1 needs to process Split 1.
• Split 1 is stored on Node 1.
• Worker 1 runs on Node 1.

Since the data and worker are on the same node, Worker 1 reads the split directly from local storage. This is a local read.

Step 4: How Many Tasks Read Data Locally?

A task reads data locally when the node holding its input split matches the node running its worker.
In our simulator:
A task is local when: split_home_node == worker_node

Because Task i is assigned to Worker i in this example, we can check locality by comparing the split’s home node with i % 4. After examining all 16 assignments, we find that only Tasks 1, 6, 11, and 12 read their data locally. The remaining 12 tasks must transfer their input data across the network.

Our cluster has 16 tasks and 16 workers distributed across four nodes. The scheduler assigns tasks to available workers without considering where the data is stored.

Step 5: How Much Additional Time Does a Remote Read Require?

In F01.1, we estimated the cost of reading and validating a single record. Let’s compare local and remote processing using the same reference model.

Local processing
When the record is stored on the worker’s machine:

• CPU validation: 20 µs
• Local disk read: 0.54 µs

Remote processing
When the record is stored on another machine, the worker also needs to transfer it over the network. Our model assumes a 1 Gbps network connection, which takes approximately 2.16 µs to transfer a 270-byte record.

Therefore: Total = 20 + 0.54 + 2.16 = 22.70 µs per record

Let’s compare the two.

The additional processing time for a remotely read record is:

(22.70 − 20.54) ÷ 20.54 × 100 ≈ 10.5%

In our simplified model, processing a record remotely takes approximately 10.5% longer than processing it locally.

However, this additional cost applies only to the records read remotely, not automatically to the entire job. Overall runtime also depends on task scheduling, parallel execution, and whether network transfers become a bottleneck.

Why Does This Matter?

Remote reads introduce additional work, even when the original data is already available somewhere in the cluster.
For our hourly job:

• 25.3% of records are read locally.
• 74.7% of records require remote reads.
• Approximately 4.84 GB moves between nodes during every run.

These transfers consume network capacity even though the job may still finish within its required time. This leads to the next engineering question:
Can we reduce network traffic by scheduling tasks on the nodes where their data is already stored?

We’ll explore this using data-aware task placement in the next section.

In the previous section, we saw that 12 of the 16 tasks read data from other nodes, transferring approximately 4.84 GB across the network. This happens because our scheduler assigns tasks to available workers without considering where their input data is stored. But what if the scheduler could choose workers based on data location?

Let’s compare three approaches.

1. Any-Worker Placement

With placement = any, the scheduler assigns each task to an available worker, regardless of where its input split is stored. This keeps scheduling simple, but tasks may need to read data from other nodes.
In our example:

• Only 4 of 16 tasks read their data locally.
• Approximately 4.84 GB moves across the network.
• The job finishes in 133.3 seconds.

2. Node-Aware Placement

With placement = node, the scheduler prefers workers running on the same node as the input data.
Here’s how it works in our simulator:

1. 01The scheduler identifies the node holding the task’s input split.
2. 02It checks whether a worker is available on that node.
3. 03If a local worker is available, the task runs there.
4. 04Otherwise, the scheduler waits up to 3 seconds for a local worker to become available.
5. 05If no local worker becomes available within that time, the task can run on another node.

This approach aims to move computation closer to the data instead of moving data to the computation. In our current configuration, every node stores four input splits and has four available workers. Since all workers are initially free, the scheduler can assign every task to a worker on its input’s home node.

As a result, all 16 tasks read their data locally.

3. Remote Data Access

The third scenario is different. Instead of storing input splits on the cluster nodes, the data is stored remotely in object storage. Every worker must retrieve its input over the network, regardless of which node executes the task.

In this scenario:

• 0% of records are read from local storage.
• All 6.48 GB of input data moves across the network.
• The job takes 134.8 seconds.

This scenario helps us compare the performance and cost of local versus remote data access.

With any-worker placement, 12 of the 16 tasks read their input remotely. With node-aware placement, all 16 tasks read their input locally.

But What About the Cost?

Now compare the final column.

• Any-worker placement: 2.99 units.
• Node-aware placement: 1.75 units.
• Remote object storage: 3.42 units.

Although the runtime difference is small, the monetary cost difference is substantial. In our model, the total cost has two components:
Total cost = Computing cost + Network transfer cost

Computing cost depends on the number of workers, runtime, and price per core-hour. Network transfer cost depends on how much data moves between nodes and the price per GB.
For any-worker placement:

• Computing cost: 1.78 units.
• Network transfer cost: 1.21 units.
• Total: 2.99 units.

With node-aware placement, network transfer drops to zero, reducing the total cost to 1.75 units.

The transfer rate is illustrative. Actual charges for traffic between cluster nodes depend on the infrastructure and cloud provider.

Our hourly validation job has been running successfully for months. It consistently finishes within its deadline, so the operations team has no reason to suspect a performance problem.
However, a closer look at the infrastructure bill reveals that network transfers account for a significant portion of the job’s cost.
The team compares two configurations:

• Any-worker placement: 133.3 seconds, costing 2.99 units.
• Node-aware placement: 131.4 seconds, costing 1.75 units.

Both configurations process the same data using the same number of workers. The only difference is how tasks are assigned to nodes.

Switching from any-worker placement to node-aware placement reduces the cost from 2.99 to 1.75 units.

Why Was This Problem Overlooked?

The operations team primarily monitored job completion time.
Since the job consistently met its deadline, the additional network traffic wasn’t considered a problem. But a job can perform well while still using resources inefficiently. In this case, the problem wasn’t visible in the overall runtime. It appeared in the network usage and cost breakdown. This is why performance monitoring should consider more than job duration. We should also examine how much data moves between nodes, which resources are being consumed, and whether those resources are necessary.

We’ve identified the source of the unnecessary network traffic. Tasks are being assigned to workers without considering where their input data is stored. The solution is to use node-aware placement, letting the scheduler prioritise workers on the same node as the data.

In our simulator, this requires changing only the placement strategy. Everything else remains unchanged:

• Same input dataset.
• Same 16 partitions.
• Same 16 workers.
• Same four cluster nodes.

What Changes After the Fix?

The improvement is straightforward. By executing tasks where their input data is stored, the scheduler eliminates unnecessary network transfers in this configuration. However, there’s an important limitation.
Node-aware scheduling does not always guarantee that every task will read data locally.

Let’s understand why.

What Happens When We Have More Tasks Than Workers?

So far, we’ve used 16 tasks and 16 workers. Because every worker was initially available, the scheduler could assign each task to a worker on its input’s home node. Now suppose we increase the number of partitions from 16 to 64, while keeping the same 16 workers. Each worker must process approximately four tasks over the course of the job. After the first round of tasks begins, some workers become busy. When the scheduler tries to assign another task, a worker on the input’s home node may not be available immediately.

The scheduler has two choices:

1. 01Wait for a local worker: Avoid network transfer, but potentially leave other available workers idle.
2. 02Use an available remote worker: Start processing sooner, but transfer input data across the network.

This introduces a tradeoff between waiting for locality and keeping workers busy.

Our job now has:

• 64 tasks
• 16 workers
• 4 cluster nodes
• Node-aware scheduling with a maximum wait of 3 seconds

What If We Wait Longer for a Local Worker?

Now increase the locality wait from 3 seconds to 30 seconds. This gives the scheduler more time to find an available worker on the input’s home node. In our simulation, the longer wait allows every task to execute locally.
The results are:

Interestingly, the longer wait improves both runtime and cost in this example. Why?
Although some tasks may wait longer for a local worker, avoiding remote reads saves enough time to offset that waiting. The job also avoids network transfer charges, reducing the total cost.

Does Waiting Longer Always Help?

No. Consider a different workload where one node’s workers are busy processing unusually large partitions. Other nodes may have available workers, but the scheduler continues waiting for a local worker. In that situation, waiting too long could leave computing resources idle and increase the overall runtime.

This is why data-aware schedulers must balance two competing goals:

• Data locality: Reduce network transfers by processing data close to where it is stored.
• Resource utilization: Keep available workers productive instead of waiting unnecessarily.

The best waiting period depends on the workload, partition sizes, worker availability, and network performance.

What Should We Measure?

Before choosing a placement policy, compare:

• Local read percentage: How much input data is processed on its home node?
• Network traffic: How many bytes are transferred between nodes?
• Task waiting time: How long do tasks wait for a local worker?
• Overall runtime: Does improved locality actually make the job faster?
• Total cost: Are network savings greater than any additional computing costs?

Our experiments demonstrate four important lessons about data locality and task scheduling.

1. 01Remote reads introduce additional costs: When a task processes data stored on another node, it must transfer that data over the network. This can increase both processing time and network usage.
2. 02Task placement affects data movement: A scheduler that ignores data location may transfer large amounts of data unnecessarily, even when the input is already stored within the cluster.
3. 03Node-aware scheduling can reduce network traffic: Assigning tasks to workers on the same nodes as their input data can improve locality and reduce transfer costs.
4. 04Data locality is not always guaranteed: When local workers are busy, the scheduler may need to choose between waiting for them or using available workers on other nodes.

Limitations of the Model

Our simulator makes several simplifying assumptions:

• Single copy of each input split: Every split is stored on exactly one node. Real distributed storage systems may replicate data across multiple nodes, giving the scheduler more opportunities to process it locally.
• Simplified network costs: We use a fixed transfer time per record and an illustrative charge per GB transferred. Actual network performance and pricing depend on the infrastructure.
• Simplified scheduling: The scheduler chooses between available workers using a fixed locality-wait period. Real systems consider additional factors, including resource availability and task scheduling policies.
• No data movement between processing tasks: We only measure the cost of reading input data. Moving intermediate results between workers, known as a shuffle, will be covered in F03.
• No failures or retries: Every task completes successfully. We’ll explore distributed execution failures and recovery in F04.

Understanding Different Types of Partitions

The word partition appears throughout Data Engineering, but its meaning depends on the system.

For example, Spark uses processing partitions to distribute computation, while Kafka uses topic partitions to organize event logs and maintain ordering within each partition. In lakehouse systems, table partitioning helps organize data so query engines can skip irrelevant files.

We’ll explore these concepts further in K01 and Data Modeling & Storage Design.

We’ve seen how node-aware scheduling improves data locality when input data is stored on the cluster. Let’s examine two related scenarios where data location changes the amount of network traffic.

Scenario 1: Reading All Input Data from Remote Storage

Suppose the team decides to store the entire input dataset in object storage instead of distributing it across the cluster’s local disks. This simplifies data access because every worker can retrieve its assigned input from the same storage system. However, every worker must now read its data over the network.

For our hourly validation job:

• Input dataset: 6.48 GB
• Local reads: 0%
• Network transfer: 6.48 GB
• Total runtime: 134.8 seconds
• Cost per run: 3.42 units

Compare this with node-aware placement when the data is stored locally.

In our model, reading remotely adds only 3.4 seconds to the runtime but significantly increases the total cost. This doesn’t mean object storage is a poor choice. It provides important advantages such as durability, independent storage scaling, and easier access across compute clusters. The engineering decision depends on whether those benefits justify the additional network usage and potential transfer charges.

Scenario 2: Increasing the Number of Cluster Nodes

Now suppose we increase the cluster from 4 nodes to 8 nodes, while continuing to assign tasks without considering data location. Will this improve locality?
Not necessarily. In our simplified model, each input split has one home node, and tasks are assigned without considering that location.

With four nodes, a task has approximately a 1-in-4 chance of being assigned to its input’s home node. With eight nodes, that probability becomes approximately 1 in 8, assuming tasks and split locations are distributed uniformly.

This means:

• 4 nodes: Approximately 25% local reads.
• 8 nodes: Approximately 12.5% local reads.

With eight nodes, roughly 87.5% of the input may need to move over the network under these assumptions. Adding more nodes can therefore increase network traffic when the scheduler ignores data location. The solution remains the same: use data-aware task placement wherever the storage system and workload support it.

How can we recognise when a distributed job is transferring more data than necessary?

In our simulator, look for three warning signs:

1. 01Low local-read percentage: Many tasks process data stored on other nodes, even though the input is available within the cluster.
2. 02High network transfer volume: A significant portion of the input dataset moves between machines during each run.
3. 03Unexpected transfer costs: Network charges contribute substantially to the job’s total cost despite relatively small differences in runtime.

What Does This Look Like in Production?

Similar symptoms may appear as:

• Recurring network traffic: Network usage increases whenever a particular processing job runs.
• High remote-read activity: Most tasks retrieve their input from machines other than the ones executing them.
• Unexpected infrastructure costs: Network transfer charges appear for data moving between nodes or infrastructure locations.
• Little improvement from network optimisations: Reducing transfers lowers costs but barely changes runtime because CPU processing remains the dominant cost.

For example, our hourly validation job transfers 4.84 GB between nodes while taking only 1.9 seconds longer than the fully local configuration. Looking only at runtime would make this problem difficult to notice.

What Should You Investigate?

Start by examining:

• Where the input splits are stored.
• Which nodes execute their corresponding tasks.
• What percentage of records are read locally.
• How much data moves between nodes.
• How much network traffic contributes to runtime and cost.

These measurements help determine whether changing task placement can improve efficiency.

Before running a job across multiple nodes, consider four important questions:

1. 01Where is the input data stored? Identify which nodes hold the input splits and whether multiple copies are available.
2. 02How does the scheduler assign tasks? Determine whether it considers data location or simply chooses the next available worker.
3. 03How long should a task wait for a local worker? Balance the benefits of local reads against the cost of leaving other workers idle.
4. 04What does data movement cost? Measure both the additional processing time and the monetary cost of transferring data across the network.

For example, our node-aware scheduler reduced network traffic from 4.84 GB to zero without changing the number of workers or input partitions. The job became only slightly faster, but its total cost decreased significantly in our model.

Let’s return to our hourly validation job. Currently, we have 4 nodes, and the scheduler uses any-worker placement without considering where input splits are stored. Approximately 25% of tasks read their input locally. Now suppose we increase the cluster to 8 nodes while keeping the same placement strategy. Each input split still has one home node.

Try It Yourself

When the F02 workbench is available, run:

The reference result should show:

• Local tasks: 4 of 16
• Data transferred: 4.84 GB

Now change the placement strategy:

Compare the results. With node-aware placement, all 16 tasks read their input locally in this scenario. The network transfer drops to zero, and total runtime decreases by approximately 1.9 seconds.

Note: The F02 workbench is planned. These commands are intended for use once it becomes available.

Data locality describes how close a processing task runs to the data it needs. When a task runs on the same node as its input data, it can read locally. Otherwise, it may need to transfer data across the network. In this lesson, we learned that:

1. 01Task placement affects network traffic: Assigning tasks without considering data location can result in unnecessary remote reads.
2. 02Node-aware scheduling improves locality: Running tasks on the nodes holding their input data can reduce network transfers.
3. 03Better locality doesn’t always mean faster execution: When CPU processing dominates, reducing remote reads may have only a small effect on runtime.
4. 04Network transfers can have significant costs: Moving data between machines may increase infrastructure expenses, even when the job meets its deadline.
5. 05Locality and worker utilisation must be balanced: Waiting for a local worker can save network traffic, but excessive waiting may increase runtime.

Our Hourly Validation Example

Using any-worker placement:

• 4.84 GB of data moved between nodes.
• The job completed in 133.3 seconds.
• Total cost was 2.99 units.

Switching to node-aware placement:

• Network transfer dropped to zero.
• Runtime decreased to 131.4 seconds.
• Total cost decreased to 1.75 units.

The job became only 1.9 seconds faster, but its total cost fell by approximately 41% in our illustrative billing model.

Sources. Capture: reference router 0.1, Python 3.13, 2026-10-13; fixture orders_s0_small.jsonl sha256 5ac4be13…; profiles hourly_validate, lab_cluster (learner pack); F01’s price list (page F01.1). The wait rule follows delay scheduling (M. Zaharia et al., “Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling”, EuroSys 2010), which Spark exposes as spark.locality.wait; the page’s numbers are the module’s own.