The short version
Why does training generate network traffic?
Large training jobs divide work among accelerators. The way a job is divided determines which devices exchange gradients, intermediate results and expert inputs, and how often they must wait for those exchanges. Data parallelism can synchronize gradients; pipeline parallelism transfers activations between stages; tensor parallelism exchanges partial results within layers; and expert parallelism can send token representations to different devices. These transfers usually occur on private cluster interconnects, not on the office Internet connection used to reach a hosted AI service.
Actual traffic depends on the model, framework, placement, communication library and network. The patterns below are starting points for measurement, not signatures that identify every job.
01 / Training networks
From a training step to network traffic
In a simplified training step, the model processes a batch in a forward pass and calculates a loss. Backpropagation calculates gradients describing how the loss changes with model parameters. An optimizer uses those gradients to update the parameters. Dividing the work among GPUs introduces communication wherever one device needs information from another. Training data reads and checkpoint writes are additional traffic with their own storage paths and schedules.
Multiple forms of parallelism are commonly combined. A GPU may belong to several communication groups at once, and computation and communication can overlap. PyTorch's DistributedDataParallel, for example, groups gradients into buckets so their reduction can overlap with backward computation.
Further reading: PyTorch: DistributedDataParallel.
| Approach | Illustrative exchange | What to examine |
|---|---|---|
| Data | Replica ↔ replica: gradients | Synchronization and throughput |
| Pipeline | Stage A → stage B: activations; gradients return | Stage placement and delay |
| Tensor | Devices ↔ devices: partial layer results | Frequent communication and locality |
| Expert | Token representations → selected experts → results | Uneven destinations and congestion |
Conceptual examples: actual operations and traffic depend on the job and implementation.
How work moves between devices
These simplified logical diagrams show example exchanges, not physical cables, switch paths, or measured traffic. Actual communication varies by model and implementation.
Examine Synchronization time and throughput.
Examine Stage placement and handoff delay.
Examine Communication frequency and locality.
Examine Uneven destinations and congestion.
Solid and dashed arrows distinguish exchanges where needed. On a narrow screen, scroll within a diagram to see its full width.
02 / Training networks
Data parallelism: synchronizing replicas
In basic data parallelism, each worker holds a model replica and processes a different batch of examples. Workers synchronize gradients so their replicas can be updated consistently. A common operation is AllReduce: the inputs are combined element by element and the result is made available to every participant. The training software may average the sum and may accumulate gradients across multiple batches before synchronizing.
A useful picture is a group of devices periodically exchanging gradient buckets. Communication can overlap computation rather than appearing as one transfer after the entire backward pass. Bucket sizes, group size and placement affect the observed flows. Sharded training designs use additional patterns, so do not assume every data-parallel job has a complete model on each GPU or uses the same collective sequence.
Further reading: PyTorch: Fully Sharded Data Parallel.
03 / Training networks
Pipeline parallelism: moving between stages
Pipeline parallelism assigns different groups of model layers to different stages. A stage sends activations to the next stage in the forward pass; during backpropagation, corresponding activation gradients flow in the reverse direction. Multiple microbatches can be in flight at once, subject to the pipeline schedule.
The network question is how long stage-to-stage transfers take and how consistently they complete. A long or variable handoff can leave a stage waiting even when a port's average utilization looks low. Work with the scheduler and training team to understand the placement of adjacent stages, the size and frequency of transfers, and the time visible to the job. Low utilization alone is not evidence that the network caused a slowdown.
Further reading: NVIDIA: Parallelism strategies.
04 / Training networks
Tensor parallelism: sharing a layer
Tensor parallelism divides computations and parameters within a layer among devices. The devices exchange partial results as the computation proceeds. Particular implementations may use AllReduce, AllGather or ReduceScatter; the number and size of exchanges depend on the model and configuration.
Frequent dependencies make placement important. A tightly connected scale-up domain can be a suitable place for a tensor-parallel group, but the useful boundary depends on the hardware and job. Measure the collectives and end-to-end training performance before assuming a particular placement is always best. Scale-up bandwidth, scale-out bandwidth and latency vary by system.
Further reading: NVIDIA: Parallelism strategies.
05 / Training networks
Expert parallelism: dispatching token representations
A mixture-of-experts model has expert layers in which a router selects experts for each token representation. Expert parallelism distributes experts across devices. The job dispatches representations to the selected experts, then returns their results for combination. Some implementations use All-to-All communication; actual dispatch methods and routing vary.
Traffic can be uneven: several senders may target experts on the same destination at the same time. When their combined arrival rate exceeds the capacity available toward that destination, queues can build. This many-to-one pattern is called incast. Look for congestion at the receiving port and across the path, and compare it with the timing and placement of the job. An All-to-All benchmark can help characterize the fabric, but it may not reproduce a particular model's changing traffic distribution.
Further reading: NVIDIA: Mixture of experts.
06 / Training networks
Collectives and physical paths
Collectives describe a group operation, not a physical cabling diagram. AllReduce combines corresponding values and returns the result to each participant. ReduceScatter combines values and assigns a portion to each participant; AllGather collects portions and makes the combined data available to all. ReduceScatter followed by AllGather can provide the equivalent result of AllReduce.
For an illustrative sum, if one device starts with [1, 2] and another with [3, 4], an AllReduce returns [4, 6] to both. This describes the result, not the exact packets or path. A library may use a ring, tree or another algorithm; its choices depend on its implementation, message size, topology and settings. A logical neighbor is not necessarily connected by one physical hop.
Further reading: NVIDIA: Collective operations.
07 / Training networks
What to measure
- Map the job's communication groups to actual GPU, NIC, switch and site placement. Identify scale-up links, scale-out fabric, storage connections and any inter-site paths.
- Ask the training team which operations, message sizes, group sizes and placement patterns occur, and whether communication overlaps compute. Capture the relevant software and configuration versions.
- Establish a baseline using representative collective benchmarks on the intended topology. Record both collective completion time and achieved throughput over a range of message and group sizes. Run tests only in an agreed window and account for competing workloads.
- Compare job step time and GPU activity with interface utilization, errors, discards, queue occupancy, congestion signals and tail latency where available. Sample rates and counter definitions matter; an average can hide a short burst. Correlation alone does not prove causation.
- Investigate a suspected path by comparing like-for-like placement and measurements. Check NICs, links, switch queues and scheduler decisions before attributing the problem to one device or buying capacity.
- Review the fabric's congestion-control and flow-control design against its equipment, transport and workload. Packet loss, pause behavior, reordering and congestion can each affect performance; settings and tradeoffs must be validated together. Do not assume one universal configuration.
If a job spans facilities, check which exchanges actually traverse the inter-site connection. Dataset ingestion and checkpoint replication have different timing and capacity requirements from tightly synchronized per-step communication. Distance adds propagation delay, and a high line rate alone cannot remove it.
Further reading: NVIDIA: NCCL Tests.
For the wider picture, see Where Does AI Traffic Travel? and Why AI Depends on Fiber Networks.
08 / Connecting facilities
From training traffic to inter-site capacity
Data sets, checkpoints and model files may need to move between a training site, storage location, cloud service or another data center. Start with the data that actually crosses the connection, its size, how often it moves and its deadline. A GPU fabric’s port rate is not automatically the capacity needed between facilities. In one example, NVIDIA describes a cluster architecture with multiple 400 Gbps compute network interfaces per server; those interfaces are part of that particular cluster design, not a specification for every inter-site connection. NVIDIA’s network architecture describes the compute, storage and customer networks separately.
Meta describes the time spent ingesting and moving large data sets between regions and the effect that slow storage access can have on training. Its account is an example of one operator’s infrastructure; actual transfer sizes, paths and deadlines vary by workload. Read Meta’s AI storage account.
A 100 terabyte (TB) transfer contains 800 trillion bits using decimal units. At a sustained rate equal to the full connection capacity, the ideal minimum transfer times are:
| Connection capacity | Ideal minimum time |
|---|---|
| 100 Gbps | 2 hours 13 minutes 20 seconds |
| 400 Gbps | 33 minutes 20 seconds |
| 800 Gbps | 16 minutes 40 seconds |
| 1.6 Tbps | 8 minutes 20 seconds |
These estimates assume all capacity is available to the transfer for its full duration. Protocol overhead, competing traffic, retries, storage performance and endpoint limits can increase the time. If the application instead sustains 80 Gbps of useful throughput, the same transfer takes about 2 hours 47 minutes. To move 100 TB in one hour requires about 222 Gbps of useful average throughput, before allowing for overhead or other traffic. The transfer size and deadlines here are examples, not a recommendation for a particular job.
For an interactive 10 TB example with selectable connection rates, use the transfer estimator in Where Does AI Traffic Travel? Measure a representative transfer from sender to receiver and account for simultaneous jobs, the available backup capacity and any verification required after transfer.
Where Stealth connectivity can fit
Transparent LAN provides private Ethernet between eligible locations; published capacity options run from 100 Gbps to 1.6 Tbps. Dark Fiber allows the customer to supply and operate compatible optical equipment. For cloud destinations, Cloud Connect offers connectivity options; confirm the provider, region, handoff, routing and any access required for private cloud resources. Availability, route and service scope are confirmed for each project.
For a training job that exchanges data continuously across sites, evaluate latency, congestion, loss, placement and recovery as well as throughput. Confirm physical route diversity, failover capacity and equipment compatibility where required. A high-capacity connection alone does not guarantee training performance or establish that a particular cluster transport is supported end to end.
References
Sources
These primary sources describe specific frameworks and implementations; behavior depends on the software and architecture in use.