Root cause analysis / Investigation

api.x.ai

Reference RCA-0A72836449A8A0C0

xAI API endpoint

Measurement point: Stealth NYC, 32 Avenue of the Americas.

Published investigation. Updates run automatically.

01 / Network path

Network path.

Endpoint measurements / ICMP IPv4

Latency over time.

Successful endpoint RTT recordings from up to one hour before and after the event.

Enter time range

Times use the selected timezone; choices include recorded samples and intervals across the available chart.

Use the recordings table below to view latency values.

Successful RTT recording Event interval

04 Nov 2026 14:30 – 04 Nov 2026 17:29 UTC. Hover or tap to inspect; drag across the chart to zoom. On touchscreens, first choose Select time range. Escape or Reset zoom restores the full range. Missing recordings are gaps; this is not an application availability or packet-loss chart.

Select a recording to explore nearby path observations.

View recorded values
Successful ICMP IPv4 RTT recordings.
Recorded at (UTC)RTT
1.358 ms
1.304 ms
1.302 ms
1.356 ms
1.410 ms
1.356 ms
1.343 ms
1.294 ms
1.470 ms
1.340 ms
1.361 ms
1.380 ms
1.336 ms
1.336 ms
1.348 ms
1.477 ms
1.385 ms
1.313 ms
1.343 ms
1.341 ms
1.371 ms
1.406 ms
1.311 ms
1.322 ms
1.396 ms
1.309 ms
1.340 ms
1.390 ms
1.395 ms
1.320 ms
1.333 ms
1.308 ms
1.369 ms
1.411 ms
1.366 ms
1.349 ms
1.330 ms
1.330 ms
1.364 ms
1.375 ms
1.371 ms
1.330 ms
1.344 ms
1.351 ms
1.290 ms
1.408 ms
1.326 ms
1.449 ms
1.316 ms
1.448 ms
1.447 ms
1.365 ms
1.324 ms
1.316 ms
1.532 ms
1.328 ms
1.351 ms
1.478 ms
1.325 ms
1.421 ms
1.323 ms
1.535 ms
1.417 ms
1.338 ms
1.319 ms
1.341 ms
1.335 ms
3.601 ms
1.338 ms
1.394 ms
1.367 ms
1.341 ms
1.416 ms
1.325 ms
9.994 ms
1.300 ms
1.441 ms
1.397 ms
1.402 ms
1.831 ms
1.321 ms
1.356 ms
1.408 ms
1.475 ms
1.331 ms
1.426 ms
1.399 ms
1.354 ms
1.393 ms
1.480 ms
1.329 ms
1.365 ms
1.363 ms
1.385 ms
1.395 ms
1.363 ms
1.371 ms
3.832 ms
1.348 ms
7.887 ms
1.429 ms
1.344 ms
1.379 ms
1.381 ms
1.339 ms
1.371 ms
1.395 ms
1.336 ms
1.478 ms
1.442 ms
1.449 ms
1.313 ms
1.495 ms
1.504 ms
1.392 ms
1.331 ms
1.380 ms
1.368 ms
2.209 ms
1.355 ms
1.345 ms
10.160 ms
1.455 ms
1.405 ms
1.471 ms
1.322 ms
1.409 ms
2.271 ms
1.333 ms
1.381 ms
3.132 ms
7.640 ms
1.375 ms
1.410 ms
1.353 ms
1.444 ms
1.358 ms
1.371 ms
1.954 ms
1.429 ms
1.359 ms
4.200 ms
10.153 ms
1.395 ms
1.361 ms
1.373 ms
1.327 ms
1.340 ms
1.378 ms
2.129 ms
1.435 ms
1.594 ms
1.410 ms
1.344 ms
1.341 ms
1.894 ms
4.707 ms
1.381 ms
5.928 ms
1.508 ms
9.012 ms
2.519 ms
1.322 ms
1.376 ms
1.408 ms
1.401 ms
1.300 ms
1.343 ms
1.378 ms
1.320 ms
1.333 ms
1.401 ms
1.296 ms
1.383 ms
1.350 ms
1.324 ms
1.392 ms
8.141 ms
2.960 ms
1.401 ms
1.321 ms
1.725 ms
2.304 ms
1.353 ms
1.331 ms
1.368 ms
1.403 ms
1.515 ms
1.338 ms
8.827 ms
1.326 ms
1.368 ms
1.329 ms
1.349 ms
6.659 ms
2.762 ms
1.320 ms
1.349 ms
1.435 ms
1.401 ms
1.661 ms
1.333 ms
1.460 ms
1.341 ms
1.427 ms
1.485 ms
1.344 ms
1.506 ms
1.341 ms
8.533 ms
1.338 ms
2.406 ms
1.314 ms
1.364 ms
1.573 ms
1.432 ms
1.400 ms
1.471 ms
1.395 ms
1.453 ms
1.462 ms
1.365 ms
1.409 ms
1.338 ms
1.410 ms
1.364 ms
1.359 ms
1.347 ms
1.499 ms
1.319 ms
1.464 ms
1.661 ms
1.331 ms
1.329 ms
1.314 ms
1.463 ms
1.373 ms
1.599 ms
1.399 ms
1.414 ms
1.406 ms
1.322 ms
1.358 ms
1.395 ms
1.515 ms
1.355 ms
1.373 ms
1.653 ms
1.362 ms
1.424 ms
1.445 ms
4.415 ms
1.399 ms
2.171 ms
1.402 ms
10.753 ms
1.330 ms
1.504 ms
1.324 ms
1.374 ms
1.427 ms
1.315 ms
1.459 ms
11.894 ms
1.398 ms
1.304 ms
1.333 ms
1.349 ms
1.351 ms
1.348 ms
1.416 ms
1.392 ms
1.911 ms
2.352 ms
1.556 ms
1.374 ms
1.430 ms
1.358 ms
1.458 ms
1.485 ms
1.359 ms
1.374 ms
1.422 ms
1.427 ms
1.320 ms
1.359 ms
1.411 ms
11.817 ms
1.503 ms
1.325 ms
3.805 ms
1.372 ms
1.316 ms
1.331 ms
1.443 ms
1.372 ms
1.351 ms
1.429 ms
1.347 ms
1.388 ms
5.854 ms
8.570 ms
1.401 ms
1.463 ms
1.379 ms
10.327 ms
1.313 ms
1.370 ms
1.378 ms
1.414 ms
1.365 ms
1.385 ms
1.361 ms
1.404 ms
6.383 ms
1.596 ms
1.333 ms
1.461 ms
1.386 ms
9.763 ms
1.377 ms
1.339 ms
1.331 ms
1.394 ms
1.327 ms
1.544 ms
1.344 ms
1.559 ms
1.338 ms
1.338 ms
1.328 ms
1.522 ms
1.343 ms
1.387 ms
1.339 ms
1.354 ms
1.557 ms
1.317 ms
1.327 ms
1.348 ms
1.333 ms
1.362 ms
1.299 ms
1.367 ms
1.355 ms
1.354 ms
1.428 ms
1.305 ms
1.369 ms
1.465 ms
1.318 ms
1.312 ms
3.297 ms
1.330 ms
1.458 ms
1.378 ms
1.341 ms
1.405 ms
1.328 ms
Network path / recorded observations

Network path observations.

16 observations

Choose a measurement type to see the closest of the available sampled observations to the selected chart point. Each probe is separate and may follow a different path.

Before

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.121 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.272 msReply RTT AS8002 · Stealth
Before

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.211 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.254 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop4.291 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.290 msReply RTT AS13335 · Cloudflare
Before

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.131 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.180 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop0.943 msReply RTT ASN unavailable
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.167 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.180 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.184 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.514 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination3.734 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.121 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.178 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop26.952 msReply RTT ASN unavailable
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.280 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.199 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop63.562 msReply RTT ASN unavailable
  4. TTL 4 Probe hop1.594 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hopNo replyNo comparable RTT ASN unavailable
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Probe hopNo replyNo comparable RTT ASN unavailable
  8. TTL 8 Probe hopNo replyNo comparable RTT ASN unavailable
  9. TTL 9 Probe hopNo replyNo comparable RTT ASN unavailable
  10. TTL 10 Probe hopNo replyNo comparable RTT ASN unavailable
  11. TTL 11 Probe hopNo replyNo comparable RTT ASN unavailable
  12. TTL 12 Probe hopNo replyNo comparable RTT ASN unavailable
  13. TTL 13 Probe hopNo replyNo comparable RTT ASN unavailable
  14. TTL 14 Probe hopNo replyNo comparable RTT ASN unavailable
  15. TTL 15 Probe hopNo replyNo comparable RTT ASN unavailable
  16. TTL 16 Probe hopNo replyNo comparable RTT ASN unavailable
  17. TTL 17 Probe hopNo replyNo comparable RTT ASN unavailable
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.130 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.179 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop47.849 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.166 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.185 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop19.286 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.457 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Probe hopNo replyNo comparable RTT ASN unavailable
  8. TTL 8 Probe hopNo replyNo comparable RTT ASN unavailable
  9. TTL 9 Probe hopNo replyNo comparable RTT ASN unavailable
  10. TTL 10 Probe hopNo replyNo comparable RTT ASN unavailable
  11. TTL 11 Probe hopNo replyNo comparable RTT ASN unavailable
  12. TTL 12 Probe hopNo replyNo comparable RTT ASN unavailable
  13. TTL 13 Probe hopNo replyNo comparable RTT ASN unavailable
  14. TTL 14 Probe hopNo replyNo comparable RTT ASN unavailable
  15. TTL 15 Probe hopNo replyNo comparable RTT ASN unavailable
  16. TTL 16 Probe hopNo replyNo comparable RTT ASN unavailable
  17. TTL 17 Probe hopNo replyNo comparable RTT ASN unavailable
During

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.488 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.261 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop15.642 msReply RTT ASN unavailable
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.188 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.218 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop4.069 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination18.471 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.131 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.179 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop24.164 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.156 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.190 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop23.953 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.652 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination1.889 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.181 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.206 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop35.434 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.185 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.182 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop20.332 msReply RTT ASN unavailable
  4. TTL 4 Probe hop1.547 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop8.859 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination19.543 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.128 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.178 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop27.024 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.343 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.313 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop104.078 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.495 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.359 msReply RTT AS13335 · Cloudflare

RTT is the round trip to each responding hop, not the delay of the preceding link. TTL order may omit routers and does not prove direct links. A missing reply is not proof of packet loss. ASN labels are dated address associations; they do not establish the forwarding path or the operator at the event time. Up to three recorded traces per method and period are selected across the archive. Where no individual trace is available, a comparison may summarize a window; MTR observations aggregate multiple probes. Address aliases are independent across measurement types.

Same method / same recorded address

Compare the traces.

Compare one selected trace from before the event with one during it. Values are individual replies, not a statistical comparison.

ICMP traceroute · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.131 msAS8002 · Stealth0.121 msAS8002 · StealthSame responder
20.180 msAS8002 · Stealth0.178 msAS8002 · StealthSame responder
30.943 ms26.952 msSame responder
TCP traceroute / 443 · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.167 msAS8002 · Stealth0.280 msAS8002 · StealthSame responder
20.180 msAS8002 · Stealth0.199 msAS8002 · StealthSame responder
31.184 ms63.562 msSame responder
4No reply1.594 msAS13335 · CloudflareNot established
51.514 msAS13335 · CloudflareNo replyNot established
63.734 msAS13335 · CloudflareNo replyNot established
7Not recordedNo replyNot established
8Not recordedNo replyNot established
9Not recordedNo replyNot established
10Not recordedNo replyNot established
11Not recordedNo replyNot established
12Not recordedNo replyNot established
13Not recordedNo replyNot established
14Not recordedNo replyNot established
15Not recordedNo replyNot established
16Not recordedNo replyNot established
17Not recordedNo replyNot established
TCP traceroute / 443 · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.211 msAS8002 · Stealth0.166 msAS8002 · StealthSame responder
20.254 msAS8002 · Stealth0.185 msAS8002 · StealthSame responder
3No reply19.286 msAS13335 · CloudflareNot established
4No replyNo replyNot established
54.291 msAS13335 · Cloudflare1.457 msAS13335 · CloudflareDifferent responders
61.290 msAS13335 · CloudflareNo replyNot established
7Not recordedNo replyNot established
8Not recordedNo replyNot established
9Not recordedNo replyNot established
10Not recordedNo replyNot established
11Not recordedNo replyNot established
12Not recordedNo replyNot established
13Not recordedNo replyNot established
14Not recordedNo replyNot established
15Not recordedNo replyNot established
16Not recordedNo replyNot established
17Not recordedNo replyNot established

RTTs include the return path. Different responders, missing replies and independent probe flows limit what a change can localize. MTR snapshots remain separate in the replay.

Event timeline.

Latency anomaly. Times shown are UTC.

Event opened
UTC
Last event observation
UTC
Recovery observed
UTC

Recorded recovery describes this event, not current reachability.

02 / Assessment

Assessment.

Investigation stateRecovered
Path assessment

Elevated latency observed.

Endpoint round-trip time was elevated during part of the event. The available path observations do not identify where the increase occurred.

What we observed

  • A sustained increase in measured round trip time was observed.
  • Stored traces were compared using matching source and destination addresses.

Recording counts describe collected measurements, not application availability or packet loss. Exact packet paths and the return path remain unverified.

Measurements at a glance.

Baseline RTT1.355 msUsual round trip before the event
Last recorded event RTT1.414 msLast successful event recording
Event p95 RTT7.665 ms95th percentile of successful recordings
Change from baseline+0.059 msLast event RTT compared with the baseline
Successful / expected recordings120 / 120Recorded endpoint samples during the event

RTT is round trip time; p95 describes the upper range of successful recordings. Endpoint measurements differ from individual trace replies and are not an uptime percentage.

Record sources and timestamps
Analysis updated
UTC
Public snapshot published
UTC
Public observation window
– UTC

Event and source timestamps are UTC. The latency chart uses its selected timezone. Page refresh times are separate from measurement timestamps.

03 / Method and limits

Method and limits.

How to interpret the sampled path and its limits

Corroborating the path.

Comparisons use matching destinations before and during the event. ICMP traces use echo probes. TCP/443 traces send SYN probes to port 443 only; they do not complete a TCP handshake or perform a TLS or HTTP exchange.

The Stealth gNMI interface verifies the active route-table choice and maps its primary next hop to the Stealth logical interface, configured attachment and, where applicable, the known LAG member set at each snapshot. Stable Stealth replies, together with mapped interface evidence showing no corresponding errors or sustained capacity pressure, reduce the likelihood of a Stealth forwarding issue. An 80% load warning warrants investigation; it does not prove congestion. Lower interval averages can conceal bursts or queue pressure. Optical readings assess the physical link, not congestion.

What remains unseen.

Probe reply handling can differ from packet forwarding. A slow intermediate reply that disappears at later hops does not locate a forwarding problem. A destination that does not respond to a trace is not thereby shown to be unavailable.

Stealth does not use ECMP in this monitored scope. The route snapshot does not identify which physical LAG member carried a particular independent probe, attest line-card forwarding at that packet instant, prove the exact application packet flow or observe the return path. RTT includes the return path, which may differ. No remote reverse traceroute is available. Tunnels and silent routers may hide parts of the path. Address registration alone does not establish the forwarding operator or responsibility.

References: ICMP reply handling / RFC 1812 · Round trip measurements / RFC 2681 · Full methodology