Root cause analysis / Investigation

api.x.ai

Reference RCA-40AAC4F3C122EC68

xAI API endpoint

Measurement point: Stealth NYC, 32 Avenue of the Americas.

Published investigation. Updates run automatically.

01 / Network path

Network path.

Endpoint measurements / ICMP IPv4

Latency over time.

Successful endpoint RTT recordings from up to one hour before and after the event.

Enter time range

Times use the selected timezone; choices include recorded samples and intervals across the available chart.

Use the recordings table below to view latency values.

Successful RTT recording Event interval

04 Nov 2026 19:00 – 04 Nov 2026 21:29 UTC. Hover or tap to inspect; drag across the chart to zoom. On touchscreens, first choose Select time range. Escape or Reset zoom restores the full range. Missing recordings are gaps; this is not an application availability or packet-loss chart.

Select a recording to explore nearby path observations.

View recorded values
Successful ICMP IPv4 RTT recordings.
Recorded at (UTC)RTT
1.380 ms
1.356 ms
1.359 ms
1.491 ms
1.359 ms
1.322 ms
1.329 ms
1.364 ms
1.339 ms
1.349 ms
1.654 ms
1.313 ms
10.860 ms
1.356 ms
5.491 ms
9.049 ms
1.395 ms
2.289 ms
1.388 ms
1.380 ms
1.439 ms
13.891 ms
1.371 ms
1.339 ms
1.303 ms
1.405 ms
1.356 ms
1.332 ms
1.405 ms
1.354 ms
1.588 ms
1.347 ms
1.365 ms
1.450 ms
1.429 ms
1.478 ms
1.347 ms
1.509 ms
1.373 ms
1.339 ms
1.388 ms
1.438 ms
1.497 ms
1.347 ms
1.356 ms
1.399 ms
1.465 ms
1.368 ms
1.317 ms
1.449 ms
1.407 ms
1.387 ms
1.419 ms
1.445 ms
4.060 ms
1.388 ms
1.343 ms
1.446 ms
1.439 ms
1.358 ms
1.404 ms
1.364 ms
1.346 ms
1.342 ms
1.358 ms
1.336 ms
1.413 ms
1.364 ms
1.399 ms
5.198 ms
1.390 ms
1.391 ms
1.357 ms
1.381 ms
1.427 ms
1.756 ms
1.468 ms
1.375 ms
1.391 ms
1.346 ms
1.317 ms
1.394 ms
1.352 ms
1.354 ms
1.397 ms
1.324 ms
1.422 ms
1.387 ms
1.386 ms
1.395 ms
1.370 ms
1.350 ms
1.422 ms
1.308 ms
1.399 ms
1.395 ms
1.412 ms
1.538 ms
6.218 ms
1.428 ms
1.439 ms
1.366 ms
1.375 ms
1.593 ms
1.403 ms
1.392 ms
13.950 ms
9.279 ms
1.352 ms
1.509 ms
1.326 ms
1.365 ms
1.442 ms
1.337 ms
1.380 ms
1.474 ms
1.406 ms
1.332 ms
1.417 ms
1.386 ms
1.389 ms
1.320 ms
1.370 ms
1.522 ms
1.362 ms
3.361 ms
16.282 ms
1.361 ms
1.338 ms
1.334 ms
1.304 ms
1.409 ms
1.341 ms
1.302 ms
1.359 ms
1.440 ms
1.359 ms
12.518 ms
1.367 ms
1.366 ms
7.222 ms
1.353 ms
1.386 ms
1.401 ms
1.333 ms
1.348 ms
1.336 ms
1.408 ms
5.190 ms
1.565 ms
1.489 ms
1.368 ms
1.359 ms
1.473 ms
1.349 ms
1.381 ms
1.410 ms
1.375 ms
1.315 ms
1.358 ms
1.339 ms
1.362 ms
1.484 ms
1.340 ms
1.366 ms
1.337 ms
1.337 ms
1.495 ms
1.367 ms
1.414 ms
1.358 ms
1.367 ms
1.380 ms
1.353 ms
1.310 ms
1.373 ms
1.358 ms
1.370 ms
1.348 ms
1.399 ms
1.355 ms
1.406 ms
1.360 ms
1.348 ms
1.415 ms
1.395 ms
1.341 ms
1.335 ms
1.353 ms
1.329 ms
1.446 ms
1.312 ms
1.305 ms
1.449 ms
1.393 ms
1.373 ms
1.468 ms
1.333 ms
1.542 ms
1.385 ms
1.365 ms
1.345 ms
1.422 ms
1.338 ms
1.375 ms
1.387 ms
1.395 ms
9.819 ms
1.432 ms
1.322 ms
1.310 ms
1.360 ms
1.327 ms
1.342 ms
1.337 ms
1.321 ms
1.291 ms
1.390 ms
1.331 ms
1.459 ms
1.880 ms
4.918 ms
1.459 ms
1.399 ms
1.345 ms
1.469 ms
1.538 ms
1.514 ms
1.345 ms
1.503 ms
1.725 ms
1.314 ms
1.356 ms
1.472 ms
1.344 ms
1.437 ms
1.340 ms
1.303 ms
1.414 ms
1.373 ms
1.433 ms
1.350 ms
1.397 ms
1.313 ms
1.578 ms
2.030 ms
1.361 ms
1.319 ms
1.356 ms
1.321 ms
1.305 ms
3.532 ms
1.364 ms
1.331 ms
1.398 ms
1.392 ms
1.338 ms
1.368 ms
1.380 ms
1.351 ms
1.471 ms
1.494 ms
1.349 ms
1.359 ms
1.412 ms
1.526 ms
1.327 ms
1.448 ms
1.331 ms
1.330 ms
1.357 ms
1.329 ms
1.351 ms
1.349 ms
1.363 ms
1.520 ms
1.413 ms
1.468 ms
1.401 ms
6.694 ms
1.321 ms
1.349 ms
1.419 ms
1.395 ms
1.359 ms
1.334 ms
1.342 ms
1.371 ms
1.291 ms
1.341 ms
1.529 ms
1.419 ms
1.412 ms
1.364 ms
1.471 ms
1.336 ms
1.339 ms
1.319 ms
1.343 ms
Network path / recorded observations

Network path observations.

17 observations

Choose a measurement type to see the closest of the available sampled observations to the selected chart point. Each probe is separate and may follow a different path.

Before

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.131 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.278 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop12.956 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.141 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.188 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop25.042 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.589 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination2.191 msReply RTT AS13335 · Cloudflare
Before

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.132 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.215 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop2.224 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.306 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.738 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.483 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.204 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination2.946 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.143 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.191 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop19.612 msReply RTT ASN unavailable
  4. TTL 4 Probe hop14.434 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop16.947 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.650 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.127 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.192 msReply RTT AS8002 · Stealth
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.158 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.185 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop18.788 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.584 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.155 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.187 msReply RTT AS8002 · Stealth
During

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.204 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.274 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.058 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.508 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.121 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.182 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop21.729 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.379 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.187 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop4.381 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop43.551 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination18.934 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.113 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.222 msReply RTT AS8002 · Stealth
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.185 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.178 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hop5.604 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop15.146 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination0.891 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.133 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.176 msReply RTT AS8002 · Stealth
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.227 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.283 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hopNo replyNo comparable RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.425 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination13.054 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.136 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.217 msReply RTT AS8002 · Stealth
After

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.193 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.250 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.459 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.151 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.837 msReply RTT AS13335 · Cloudflare

RTT is the round trip to each responding hop, not the delay of the preceding link. TTL order may omit routers and does not prove direct links. A missing reply is not proof of packet loss. ASN labels are dated address associations; they do not establish the forwarding path or the operator at the event time. Up to three recorded traces per method and period are selected across the archive. Where no individual trace is available, a comparison may summarize a window; MTR observations aggregate multiple probes. Address aliases are independent across measurement types.

Same method / same recorded address

Compare the traces.

Compare one selected trace from before the event with one during it. Values are individual replies, not a statistical comparison.

ICMP traceroute · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.132 msAS8002 · Stealth0.127 msAS8002 · StealthSame responder
20.215 msAS8002 · Stealth0.192 msAS8002 · StealthSame responder
32.224 msAS13335 · CloudflareNot recordedNot established
ICMP traceroute · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.131 msAS8002 · Stealth0.155 msAS8002 · StealthSame responder
20.278 msAS8002 · Stealth0.187 msAS8002 · StealthSame responder
312.956 msAS13335 · CloudflareNot recordedNot established
TCP traceroute / 443 · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.143 msAS8002 · Stealth0.379 msAS8002 · StealthSame responder
20.191 msAS8002 · Stealth0.187 msAS8002 · StealthSame responder
319.612 ms4.381 msAS13335 · CloudflareSame responder
414.434 msAS13335 · CloudflareNo replyNot established
516.947 msAS13335 · Cloudflare43.551 msAS13335 · CloudflareDifferent responders
61.650 msAS13335 · Cloudflare18.934 msAS13335 · CloudflareSame responder
TCP traceroute / 443 · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.306 msAS8002 · Stealth0.158 msAS8002 · StealthSame responder
20.738 msAS8002 · Stealth0.185 msAS8002 · StealthSame responder
31.483 msAS13335 · CloudflareNo replyNot established
4No replyNo replyNot established
52.204 msAS13335 · Cloudflare18.788 msAS13335 · CloudflareDifferent responders
6No reply1.584 msAS13335 · CloudflareNot established
72.946 msAS13335 · CloudflareNot recordedNot established

RTTs include the return path. Different responders, missing replies and independent probe flows limit what a change can localize. MTR snapshots remain separate in the replay.

Event timeline.

Latency anomaly. Times shown are UTC.

Event opened
UTC
Last event observation
UTC
Recovery observed
UTC

Recorded recovery describes this event, not current reachability.

02 / Assessment

Assessment.

Investigation stateRecovered
Path assessment

Elevated latency observed.

Endpoint round-trip time was elevated during part of the event. The available path observations do not identify where the increase occurred.

What we observed

  • A sustained increase in measured round trip time was observed.
  • Stored traces were compared using matching source and destination addresses.

Recording counts describe collected measurements, not application availability or packet loss. Exact packet paths and the return path remain unverified.

Measurements at a glance.

Baseline RTT1.357 msUsual round trip before the event
Last recorded event RTT1.399 msLast successful event recording
Event p95 RTT5.292 ms95th percentile of successful recordings
Change from baseline+0.042 msLast event RTT compared with the baseline
Successful / expected recordings60 / 60Recorded endpoint samples during the event

RTT is round trip time; p95 describes the upper range of successful recordings. Endpoint measurements differ from individual trace replies and are not an uptime percentage.

Record sources and timestamps
Analysis updated
UTC
Public snapshot published
UTC
Public observation window
– UTC

Event and source timestamps are UTC. The latency chart uses its selected timezone. Page refresh times are separate from measurement timestamps.

03 / Method and limits

Method and limits.

How to interpret the sampled path and its limits

Corroborating the path.

Comparisons use matching destinations before and during the event. ICMP traces use echo probes. TCP/443 traces send SYN probes to port 443 only; they do not complete a TCP handshake or perform a TLS or HTTP exchange.

The Stealth gNMI interface verifies the active route-table choice and maps its primary next hop to the Stealth logical interface, configured attachment and, where applicable, the known LAG member set at each snapshot. Stable Stealth replies, together with mapped interface evidence showing no corresponding errors or sustained capacity pressure, reduce the likelihood of a Stealth forwarding issue. An 80% load warning warrants investigation; it does not prove congestion. Lower interval averages can conceal bursts or queue pressure. Optical readings assess the physical link, not congestion.

What remains unseen.

Probe reply handling can differ from packet forwarding. A slow intermediate reply that disappears at later hops does not locate a forwarding problem. A destination that does not respond to a trace is not thereby shown to be unavailable.

Stealth does not use ECMP in this monitored scope. The route snapshot does not identify which physical LAG member carried a particular independent probe, attest line-card forwarding at that packet instant, prove the exact application packet flow or observe the return path. RTT includes the return path, which may differ. No remote reverse traceroute is available. Tunnels and silent routers may hide parts of the path. Address registration alone does not establish the forwarding operator or responsibility.

References: ICMP reply handling / RFC 1812 · Round trip measurements / RFC 2681 · Full methodology