Root cause analysis / Investigation

api.x.ai

Reference RCA-1CA18BBB2B33B0F0

xAI API endpoint

Measurement point: Stealth NYC, 32 Avenue of the Americas.

Published investigation. Updates run automatically.

01 / Network path

Network path.

Endpoint measurements / ICMP IPv4

Latency over time.

Successful endpoint RTT recordings from up to one hour before and after the event. Points are sampled from the retained history.

Enter time range

Times use the selected timezone; choices include recorded samples and intervals across the available chart.

Use the recordings table below to view latency values.

Successful RTT recording Event interval

02 Nov 2026 18:10 – 02 Nov 2026 21:14 UTC. Hover or tap to inspect; drag across the chart to zoom. On touchscreens, first choose Select time range. Escape or Reset zoom restores the full range. Missing recordings are gaps; this is not an application availability or packet-loss chart.

Select a recording to explore nearby path observations.

View recorded values
Successful ICMP IPv4 RTT recordings; displayed points are a sample of the retained records.
Recorded at (UTC)RTT
1.363 ms
1.413 ms
1.330 ms
21.454 ms
1.359 ms
1.318 ms
1.351 ms
1.364 ms
1.331 ms
1.353 ms
1.393 ms
1.359 ms
1.354 ms
1.332 ms
1.404 ms
1.456 ms
1.383 ms
1.390 ms
12.824 ms
1.339 ms
1.337 ms
5.148 ms
1.333 ms
1.431 ms
1.372 ms
1.442 ms
1.381 ms
6.460 ms
1.506 ms
1.539 ms
2.053 ms
1.359 ms
1.347 ms
1.459 ms
1.427 ms
1.365 ms
1.692 ms
1.342 ms
1.352 ms
1.361 ms
1.308 ms
1.370 ms
1.418 ms
1.385 ms
3.755 ms
2.000 ms
1.347 ms
1.316 ms
1.334 ms
1.331 ms
2.190 ms
1.433 ms
1.349 ms
1.412 ms
1.340 ms
1.301 ms
1.397 ms
1.438 ms
1.335 ms
1.359 ms
1.362 ms
1.330 ms
1.314 ms
1.320 ms
1.365 ms
1.347 ms
1.397 ms
1.311 ms
1.346 ms
1.438 ms
6.035 ms
1.336 ms
1.492 ms
1.375 ms
1.329 ms
8.033 ms
1.364 ms
1.384 ms
1.320 ms
1.356 ms
1.340 ms
1.393 ms
1.468 ms
1.505 ms
1.408 ms
1.566 ms
1.340 ms
1.357 ms
1.459 ms
1.493 ms
1.375 ms
7.209 ms
1.325 ms
1.337 ms
1.417 ms
1.338 ms
1.498 ms
1.370 ms
1.374 ms
1.353 ms
1.404 ms
1.414 ms
1.950 ms
1.359 ms
2.735 ms
7.987 ms
1.316 ms
1.350 ms
1.327 ms
1.331 ms
2.056 ms
1.340 ms
1.350 ms
1.333 ms
1.340 ms
1.329 ms
1.291 ms
1.350 ms
1.342 ms
1.412 ms
9.468 ms
10.288 ms
1.429 ms
1.421 ms
1.393 ms
1.314 ms
1.370 ms
1.322 ms
1.382 ms
2.717 ms
2.051 ms
1.355 ms
9.762 ms
1.327 ms
3.084 ms
1.488 ms
1.305 ms
2.994 ms
1.501 ms
1.476 ms
4.395 ms
1.991 ms
1.357 ms
5.176 ms
1.853 ms
1.357 ms
1.314 ms
1.316 ms
1.315 ms
1.495 ms
1.424 ms
1.495 ms
1.438 ms
1.411 ms
1.385 ms
11.087 ms
1.862 ms
1.347 ms
1.411 ms
1.400 ms
1.474 ms
1.338 ms
1.392 ms
1.361 ms
1.344 ms
1.343 ms
1.452 ms
1.401 ms
12.630 ms
1.878 ms
1.382 ms
2.981 ms
3.581 ms
2.622 ms
1.318 ms
1.361 ms
11.362 ms
11.411 ms
1.480 ms
1.327 ms
1.354 ms
1.325 ms
1.448 ms
1.401 ms
1.549 ms
1.383 ms
1.437 ms
1.341 ms
1.412 ms
1.304 ms
2.482 ms
1.335 ms
1.347 ms
1.531 ms
1.380 ms
1.306 ms
1.426 ms
1.342 ms
1.404 ms
1.349 ms
2.358 ms
1.466 ms
1.381 ms
1.351 ms
1.368 ms
1.391 ms
5.608 ms
1.406 ms
1.324 ms
1.382 ms
1.352 ms
1.369 ms
1.359 ms
1.385 ms
2.601 ms
1.361 ms
1.408 ms
1.355 ms
1.815 ms
1.731 ms
1.381 ms
1.328 ms
1.732 ms
1.323 ms
1.380 ms
1.350 ms
1.328 ms
1.460 ms
1.378 ms
1.356 ms
1.402 ms
1.338 ms
1.347 ms
1.481 ms
1.354 ms
1.342 ms
1.464 ms
1.393 ms
1.374 ms
1.419 ms
1.361 ms
1.389 ms
1.390 ms
1.317 ms
1.345 ms
1.324 ms
1.364 ms
1.335 ms
1.308 ms
1.395 ms
1.451 ms
1.334 ms
1.337 ms
1.386 ms
1.384 ms
1.371 ms
1.486 ms
1.408 ms
1.336 ms
1.375 ms
1.389 ms
1.338 ms
12.143 ms
1.402 ms
2.589 ms
1.358 ms
1.379 ms
1.346 ms
1.389 ms
1.443 ms
1.458 ms
1.318 ms
1.333 ms
1.353 ms
1.349 ms
1.400 ms
1.364 ms
1.419 ms
1.324 ms
1.397 ms
1.336 ms
1.362 ms
1.444 ms
1.442 ms
1.324 ms
1.373 ms
1.341 ms
1.386 ms
1.346 ms
1.343 ms
1.353 ms
1.312 ms
1.342 ms
1.310 ms
1.354 ms
1.337 ms
1.405 ms
1.323 ms
1.416 ms
1.518 ms
1.423 ms
1.326 ms
1.827 ms
1.386 ms
1.328 ms
1.500 ms
1.369 ms
2.096 ms
1.349 ms
1.342 ms
1.355 ms
1.360 ms
1.304 ms
1.389 ms
1.435 ms
1.391 ms
1.353 ms
1.342 ms
1.358 ms
1.358 ms
1.309 ms
1.426 ms
1.331 ms
1.362 ms
1.469 ms
1.361 ms
1.546 ms
1.342 ms
1.358 ms
1.310 ms
1.324 ms
1.344 ms
1.304 ms
1.347 ms
1.433 ms
1.372 ms
1.319 ms
1.356 ms
1.328 ms
1.351 ms
1.342 ms
1.360 ms
1.348 ms
1.314 ms
1.310 ms
1.322 ms
1.320 ms
1.330 ms
1.329 ms
1.428 ms
1.318 ms
1.375 ms
1.355 ms
3.037 ms
1.398 ms
4.956 ms
1.574 ms
4.356 ms
1.334 ms
Network path / recorded observations

Network path observations.

17 observations

Choose a measurement type to see the closest of the available sampled observations to the selected chart point. Each probe is separate and may follow a different path.

Before

ICMP traceroute · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.529 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.203 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop17.141 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hop6.011 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop17.339 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.410 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.163 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.192 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop17.441 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.093 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination17.904 msReply RTT AS13335 · Cloudflare
Before

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.124 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.176 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop17.136 msReply RTT ASN unavailable
Before

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.231 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.191 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop16.769 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.012 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination1.907 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.216 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.184 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop53.851 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop21.128 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination20.444 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.137 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.191 msReply RTT AS8002 · Stealth
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.228 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.284 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop7.429 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination0.886 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.136 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.190 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop18.628 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.231 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.192 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop16.870 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.505 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.405 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.126 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.184 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop2.313 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.165 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.195 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop2.332 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop4.726 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.678 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.129 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.261 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.098 msReply RTT AS13335 · Cloudflare
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.139 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.184 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop3.560 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop3.425 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.239 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.135 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.179 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop18.317 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hop1.626 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop34.050 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination12.473 msReply RTT AS13335 · Cloudflare
After

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.158 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.204 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop19.500 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop20.404 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination1.908 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.129 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.176 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.613 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.150 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.189 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.777 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination1.562 msReply RTT AS13335 · Cloudflare

RTT is the round trip to each responding hop, not the delay of the preceding link. TTL order may omit routers and does not prove direct links. A missing reply is not proof of packet loss. ASN labels are dated address associations; they do not establish the forwarding path or the operator at the event time. Up to three recorded traces per method and period are selected across the archive. Where no individual trace is available, a comparison may summarize a window; MTR observations aggregate multiple probes. Address aliases are independent across measurement types.

Same method / same recorded address

Compare the traces.

Compare one selected trace from before the event with one during it. Values are individual replies, not a statistical comparison.

ICMP traceroute · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.124 msAS8002 · Stealth0.126 msAS8002 · StealthSame responder
20.176 msAS8002 · Stealth0.184 msAS8002 · StealthSame responder
317.136 ms2.313 msAS13335 · CloudflareSame responder
ICMP traceroute · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.529 msAS8002 · Stealth0.137 msAS8002 · StealthSame responder
20.203 msAS8002 · Stealth0.191 msAS8002 · StealthSame responder
317.141 msAS13335 · CloudflareNot recordedNot established
46.011 msAS13335 · CloudflareNot recordedNot established
517.339 msAS13335 · CloudflareNot recordedNot established
61.410 msAS13335 · CloudflareNot recordedNot established
TCP traceroute / 443 · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.163 msAS8002 · Stealth0.231 msAS8002 · StealthSame responder
20.192 msAS8002 · Stealth0.192 msAS8002 · StealthSame responder
317.441 msAS13335 · Cloudflare16.870 msAS13335 · CloudflareSame responder
4No replyNo replyNot established
52.093 msAS13335 · Cloudflare1.505 msAS13335 · CloudflareDifferent responders
617.904 msAS13335 · Cloudflare1.405 msAS13335 · CloudflareSame responder
TCP traceroute / 443 · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.216 msAS8002 · Stealth0.228 msAS8002 · StealthSame responder
20.184 msAS8002 · Stealth0.284 msAS8002 · StealthSame responder
353.851 msNo replyNot established
4No replyNo replyNot established
521.128 msAS13335 · Cloudflare7.429 msAS13335 · CloudflareDifferent responders
620.444 msAS13335 · Cloudflare0.886 msAS13335 · CloudflareSame responder

RTTs include the return path. Different responders, missing replies and independent probe flows limit what a change can localize. MTR snapshots remain separate in the replay.

Event timeline.

Latency anomaly. Times shown are UTC.

Event opened
UTC
Last event observation
UTC
Recovery observed
UTC

Recorded recovery describes this event, not current reachability.

02 / Assessment

Assessment.

Investigation stateRecovered
Path assessment

Elevated latency observed.

Endpoint round-trip time was elevated during part of the event. The available path observations do not identify where the increase occurred.

What we observed

  • A sustained increase in measured round trip time was observed.
  • Stored traces were compared using matching source and destination addresses.

Recording counts describe collected measurements, not application availability or packet loss. Exact packet paths and the return path remain unverified.

Measurements at a glance.

Baseline RTT1.350 msUsual round trip before the event
Last recorded event RTT1.419 msLast successful event recording
Event p95 RTT9.053 ms95th percentile of successful recordings
Change from baseline+0.069 msLast event RTT compared with the baseline
Successful / expected recordings130 / 130Recorded endpoint samples during the event

RTT is round trip time; p95 describes the upper range of successful recordings. Endpoint measurements differ from individual trace replies and are not an uptime percentage.

Record sources and timestamps
Analysis updated
UTC
Public snapshot published
UTC
Public observation window
– UTC

Event and source timestamps are UTC. The latency chart uses its selected timezone. Page refresh times are separate from measurement timestamps.

03 / Method and limits

Method and limits.

How to interpret the sampled path and its limits

Corroborating the path.

Comparisons use matching destinations before and during the event. ICMP traces use echo probes. TCP/443 traces send SYN probes to port 443 only; they do not complete a TCP handshake or perform a TLS or HTTP exchange.

The Stealth gNMI interface verifies the active route-table choice and maps its primary next hop to the Stealth logical interface, configured attachment and, where applicable, the known LAG member set at each snapshot. Stable Stealth replies, together with mapped interface evidence showing no corresponding errors or sustained capacity pressure, reduce the likelihood of a Stealth forwarding issue. An 80% load warning warrants investigation; it does not prove congestion. Lower interval averages can conceal bursts or queue pressure. Optical readings assess the physical link, not congestion.

What remains unseen.

Probe reply handling can differ from packet forwarding. A slow intermediate reply that disappears at later hops does not locate a forwarding problem. A destination that does not respond to a trace is not thereby shown to be unavailable.

Stealth does not use ECMP in this monitored scope. The route snapshot does not identify which physical LAG member carried a particular independent probe, attest line-card forwarding at that packet instant, prove the exact application packet flow or observe the return path. RTT includes the return path, which may differ. No remote reverse traceroute is available. Tunnels and silent routers may hide parts of the path. Address registration alone does not establish the forwarding operator or responsibility.

References: ICMP reply handling / RFC 1812 · Round trip measurements / RFC 2681 · Full methodology