The short version
What does the work need from your network?
AI can mean an employee asking a cloud service a question, an application processing images, or a group of servers training a model. These activities move different amounts of data across different paths. Start by identifying the work, where it runs, and how quickly it needs an answer.
Bandwidth helps determine how much data can be carried over time. Latency affects how quickly information travels across a path. Both can matter, but neither alone determines how quickly an AI service responds.
01 / The data path
Trace where information travels.
Draw a simple path from each user or application to the service and back. Include the office network, firewall, Internet or private connection, cloud region or data center, storage, and AI service. A remote employee’s VPN may send traffic through the office before it reaches a cloud service. For a closer look at the journey, see how Internet traffic reaches a cloud or AI service.
Training inside a local cluster may use the cluster network rather than your Internet connection. Uploading a training set from the office to a cloud provider may use the Internet or a separate private connection. The architecture determines which capacity to plan.
02 / Everyday AI use
What makes a response feel slow?
For an interactive service, the user’s experience can include network travel, waiting for an available service, processing the input, generating the response, and any tool or data lookups. A short text request can behave very differently from a large document, image, audio stream, or video.
Measure the time until the first response and the time to complete the task, including during busy periods. For streaming text, teams may also measure the pace of the output. Google’s inference guidance calls the initial wait time to first token; this is one useful measurement, not a complete diagnosis.
If the connection is congested, more available capacity may reduce queueing. If the delay is mainly inside the application or model service, a larger Internet circuit alone may not help. Compare observations across locations and times before changing capacity.
03 / Moving data
Work backward from the deadline.
Large data sets, model files, backups, and media can place sustained demand on a connection. Estimate how much data must cross each link, in which direction, and how long the transfer can take. Add other traffic that must continue at the same time.
Illustrative minimum for 1 decimal terabyte at the stated sustained rate. This assumes the entire rate is available and excludes overhead, contention, storage and endpoint limits, and interruptions. Actual transfers take longer.
A larger link can reduce transfer time when the rest of the path can sustain it. Check sender and receiver storage, network interfaces, firewalls, cloud-service limits, and concurrency. Use the bandwidth planning guide for sizing examples.
04 / Training & model operations
Separate the cluster from its connections.
Distributed training can exchange substantial data between machines. The network linking those machines and their storage may have very different requirements from the connection between an office and the Internet. Training data ingestion, checkpoints, and model distribution may use yet other paths.
Ask the people operating the workload which transfers occur, their frequency, deadlines, and recovery requirements. For each path, identify the source, destination, measured rate, and any shared capacity. GPU, storage, and application bottlenecks can leave a fast network underused; network congestion can also slow the work.
A business using a hosted AI application may never run a training cluster itself. Size for the work it actually performs rather than assuming every AI deployment requires the same service.
05 / Performance
Measure the right delay.
Compare latency and packet loss to relevant endpoints from representative locations, then compare those observations with actual application response times. Test at different times and under normal load. Paths and service performance can change; a single ping or speed test is only a snapshot.
Higher bandwidth increases the rate at which a link can transmit data and may shorten a transfer or relieve a busy queue. It does not make signals travel faster along the same route or remove the service’s processing time. Location, routing, congestion, and the application all matter.
See latency, jitter, and packet loss for measurement techniques and their limits.
06 / Readiness
Look beyond the circuit.
The router, firewall, optics, switches, Wi-Fi, and end devices must support the intended workload. Performance with encryption or security inspection enabled may differ from an equipment port’s advertised speed. Verify the actual configuration and measure each direction.
Consider the sensitivity of prompts, documents, training data, and generated output. Follow your organization’s access, retention, privacy, and security policies when choosing services and moving data. For critical work, plan for an unavailable connection or AI service and test the recovery process.
Planning checklist
Questions for your team.
- Which AI applications do employees and systems use, and where are those services hosted?
- Which data crosses the office Internet connection, a private connection, or a local cluster?
- How many users or jobs run at once? What are the busy periods and expected growth?
- How large are the transfers, and what completion times do they require?
- What do users experience for time to first response and task completion?
- Do the equipment, storage, security services, and remote endpoints sustain the intended rates?
- Which data-handling, resilience, and recovery requirements apply?
For an initial connection estimate, see the bandwidth calculator. Validate its assumptions against the actual workload.
References
Sources & further reading.
These sources describe particular products and architectures. Use their metrics and design guidance in the context of your own application.
Inference & response
- Google Cloud: AI/ML inferenceTime to first token and other performance metrics
- Google Cloud: AI application networkingApplication traffic and service processing behavior
Training, storage & paths
- NVIDIA: Reference architecture networkingTraffic within AI clusters
- NVIDIA: Storage for AI workloadsData access, training, and checkpoints
- Google Cloud: Interconnect FAQCloud connectivity and Internet routing
Prepared by Stealth Communications, a provider of business connectivity. The appropriate design depends on the workload, network, providers, and service agreements.