The short version
One request can involve several networks
When you use an artificial intelligence (AI) service, your request travels from your device or application to the service. Behind that service, other networks connect the computers and storage doing the work. Some organizations also connect separate AI facilities to move data or coordinate computing.
Those connections do different jobs. A short request from an office does not necessarily need the same network as a training system moving large datasets between data centers. Some AI models run locally, without sending a request to a hosted service at all.
01 / Your connection
Follow a request from your business
Imagine an employee asking a hosted assistant to summarize a document. The request travels through the office network and firewall, over the Internet or an appropriate private connection, to the provider’s service endpoint—the address where the service accepts requests. The provider processes it and sends a response back. An application can make a similar request through an application programming interface (API), a defined way for software systems to exchange requests and responses.
TX: transmit, from your business to the service. RX: receive, from the service back to your business. The blocks represent packets: small units of network data.
Illustrative only: processing may involve other systems or locations. The motion does not represent measured timing.
Animation is turned off by your reduced-motion preference.
Inference is the use of an already trained model to produce an answer. The time a person waits includes network travel, any queue at the service, processing by the model, and possibly other searches or tool calls. A congested connection can matter, but buying a faster circuit does not by itself eliminate delays inside an application.
The service endpoint is not necessarily where the computing takes place. For example, Amazon Bedrock documents an option to route inference requests across regions. Which regions can process a request depends on the selected profile and its settings. Check the provider’s current documentation and your organization’s data requirements.
02 / Inside the facility
How the computers and data connect
Large AI systems can use many specialized processors, often graphics processing units (GPUs), to perform calculations. A cluster is a group of computers working together. The processors exchange intermediate results, read data, and save progress. These transfers may stay inside a provider’s private infrastructure rather than crossing your business Internet connection.
Within a computing group
Scale-up: closely connects processors working together.
Between computing groups
Scale-out: connects groups across a larger cluster.
To data and other sites
Front-end: connects computing to storage, services, and external networks.
Conceptual illustration by Stealth Communications.
Scale-up and scale-out describe relationships between computing resources, not a universal rule about how many racks they occupy. A front-end network can connect the cluster to storage and other services; its exact role varies by design. For a closer look at how these systems grow, see Why AI Depends on Fiber Networks.
Storage has its own demands. A checkpoint is a saved state that lets a training job resume from an earlier point. Many computers may read or save data at once, creating short, heavy bursts. Meta describes checkpoint-related congestion and storage delays in its own infrastructure. That example illustrates a possible bottleneck; it does not describe every AI service.
03 / Between facilities
Connecting AI sites
When an organization uses more than one data center, it may need to transfer datasets, checkpoints, model files, or application data between them. Some distributed workloads exchange information during a job as well. Their requirements depend on how often the sites communicate and how much delay the work can tolerate; distant sites do not automatically behave like one local cluster.
Dark fiber is optical fiber provided without the transmission equipment that sends light through it. The customer lights and operates the connection using its own compatible optical equipment. It can give an organization flexibility to design and expand a network between locations, subject to the available route, fiber, equipment, and agreement.
The primary route is shown in use. The alternate is an optional path.
The primary route is interrupted. An alternate can carry traffic only if that path, the equipment, capacity and failover are provisioned for it.
Two lines on a diagram do not establish physical diversity. Confirm shared ducts, entrances, equipment, and remaining capacity. Movement illustrates light on the selected path; it is not live traffic.
Animation is turned off by your reduced-motion preference; the interruption control still changes the illustration.
A managed connection is another option: the provider operates the transport and delivers an agreed service at the handoff. These choices place different responsibilities for equipment, monitoring, upgrades, and repairs on the customer and provider. Neither automatically guarantees route diversity. Ask whether proposed paths share conduits, building entrances, power, or equipment, and what capacity remains if one path fails.
Optical systems can also carry several separate light channels over a fiber. This is called wavelength division multiplexing. It can add capacity, but reach, equipment compatibility, available fiber, and the design of each route still matter. Meta’s account of its AI backbone is an example of one operator planning fiber and optical systems around its own sites; its design and capacity are not a specification for every network.
04 / Capacity planning
Work backward from the transfer
Start with the amount of data, where it begins and ends, how often it moves, and the time allowed. Average utilization alone can hide a short period when several transfers compete for the same connection.
Suppose a team wants to finish an overnight data transfer between 1:00 and 5:00 a.m. Using decimal units, 10 terabytes (TB) is 80 trillion bits. Divided evenly across four hours, that requires about 5.6 gigabits per second (Gbps) of useful data throughput. This is an illustrative window and ideal average, not an industry standard or a circuit recommendation. Protocol overhead, competing traffic, retries, equipment limits, slower storage, and backup verification can require more time or capacity.
Enable JavaScript to choose a rate. The 100 Gbps estimate is shown.
The four-hour deadline is an example, not an industry standard. Ideal estimates assume the transfer sustains the selected capacity. Overhead, competing traffic, retries, and storage speed can increase the time. The animation is illustrative, not a clock or live measurement.
Animation is turned off by your reduced-motion preference. The selector and estimates remain available.
A connection’s advertised rate is not the same as the throughput an application actually achieves. Measure a representative transfer at busy times, include the receiving system, and agree what should happen if it misses its window. If service must continue after a route fails, consider the remaining capacity as well as normal operation.
05 / Measurement
What your measurements can—and cannot—show
Latency is the time data takes to travel a path; many common tests measure a round trip. Bandwidth is the capacity of a connection, while throughput is the amount of useful data transferred over time. These are related but different measurements.
A ping or traceroute can help investigate a path to the address tested. It does not reveal every private network beyond that point or prove which facility or processor handled a request. An AI provider’s public network may also carry traffic for many services. Compare network measurements with application response times and provider status before assigning a cause.
See our guide to latency, jitter, and packet loss and how Internet traffic reaches a cloud or AI service for practical measurement context.
06 / Planning
Questions to take to your team or provider
- Which applications, users, data stores, and facilities need to communicate?
- What moves across each connection, how large are transfers, and when must they finish?
- Which response times matter to users, and where can they be measured?
- Does the work require public Internet access, private cloud connectivity, or a connection between facilities?
- Who supplies and operates the fiber, optics, monitoring, and repairs?
- What paths, entrances, and equipment are shared, and what capacity remains during a failure?
- How will capacity, power, installation, and service dates grow with the project?
For an office using hosted AI, begin with the actual application and data paths. For organizations connecting facilities, discuss locations, route options, capacity, optical requirements, and delivery timing with the relevant providers. Stealth’s Cloud Connect and Dark Fiber pages describe different service options; availability and design are confirmed for each project.
Reference
Terms to know
- AI
- Artificial intelligence: computer systems that perform tasks using trained models or other methods.
- API
- Application programming interface: a defined way for software to request a service or exchange data.
- GPU
- Graphics processing unit: a processor used for many parallel calculations, including AI work.
- Gbps
- Gigabits per second: a measure of data rate, using billions of bits per second.
- TB
- Terabyte: a measure of data size. In the example above, one terabyte means one trillion bytes; one byte contains eight bits.
References
Sources & further reading
Provider examples describe their own systems and are not Stealth service commitments. The RIPE Labs article informed this guide’s discussion of network domains and discloses the author’s commercial relationships.
Architecture & storage
- RIPE Labs: The Internet’s New FactoriesNetwork domains and the distinction between public and private traffic
- Meta: AI storage at scaleStorage, checkpoints, congestion, and processing delays
Connecting the work
- Meta: Scaling backbone connectivity for AIOne operator’s approach to connecting facilities
- Amazon Bedrock: Cross-region inferenceHow an inference profile can route requests across regions
Prepared by Stealth Communications, a provider of business connectivity.