Root cause analysis / Investigation

api.x.ai

Reference RCA-50D41631F092BDE9

xAI API endpoint

Measurement point: Stealth NYC, 32 Avenue of the Americas.

Published investigation. Updates run automatically.

01 / Network path

Network path.

Endpoint measurements / ICMP IPv4

Latency over time.

Successful endpoint RTT recordings from up to one hour before and after the event. Points are sampled from the retained history.

Enter time range

Times use the selected timezone; choices include recorded samples and intervals across the available chart.

Use the recordings table below to view latency values.

Successful RTT recording Event interval

05 Nov 2026 02:00 – 05 Nov 2026 05:09 UTC. Hover or tap to inspect; drag across the chart to zoom. On touchscreens, first choose Select time range. Escape or Reset zoom restores the full range. Missing recordings are gaps; this is not an application availability or packet-loss chart.

Select a recording to explore nearby path observations.

View recorded values
Successful ICMP IPv4 RTT recordings; displayed points are a sample of the retained records.
Recorded at (UTC)RTT
1.344 ms
1.385 ms
1.312 ms
1.355 ms
1.311 ms
1.408 ms
1.341 ms
3.424 ms
1.338 ms
1.301 ms
1.382 ms
1.334 ms
1.503 ms
1.377 ms
1.293 ms
1.329 ms
1.462 ms
1.345 ms
1.379 ms
1.330 ms
1.282 ms
1.335 ms
1.393 ms
1.367 ms
1.506 ms
1.345 ms
1.371 ms
1.506 ms
1.356 ms
1.312 ms
1.326 ms
1.343 ms
1.402 ms
1.367 ms
1.338 ms
1.345 ms
1.432 ms
1.361 ms
1.354 ms
1.305 ms
1.366 ms
1.379 ms
1.312 ms
1.320 ms
1.370 ms
1.413 ms
1.298 ms
1.378 ms
1.359 ms
1.374 ms
1.433 ms
1.481 ms
1.325 ms
1.357 ms
1.588 ms
1.355 ms
1.309 ms
1.357 ms
1.328 ms
1.366 ms
1.466 ms
1.347 ms
1.341 ms
1.836 ms
1.328 ms
1.349 ms
1.347 ms
1.392 ms
1.332 ms
1.334 ms
1.304 ms
1.405 ms
1.344 ms
1.315 ms
1.337 ms
1.431 ms
1.422 ms
1.342 ms
1.366 ms
1.491 ms
1.408 ms
1.388 ms
1.301 ms
1.360 ms
1.333 ms
1.343 ms
1.324 ms
1.353 ms
1.326 ms
1.412 ms
1.341 ms
1.325 ms
1.321 ms
1.419 ms
1.325 ms
1.441 ms
1.416 ms
1.333 ms
1.335 ms
1.373 ms
1.400 ms
8.865 ms
1.357 ms
1.309 ms
1.330 ms
1.347 ms
1.476 ms
1.327 ms
1.328 ms
1.364 ms
1.321 ms
1.371 ms
1.384 ms
1.355 ms
1.325 ms
1.384 ms
1.399 ms
1.344 ms
1.366 ms
1.394 ms
1.375 ms
1.402 ms
1.315 ms
1.367 ms
1.374 ms
5.384 ms
1.337 ms
1.395 ms
4.405 ms
1.321 ms
1.456 ms
16.297 ms
7.550 ms
3.190 ms
2.319 ms
2.559 ms
1.429 ms
3.353 ms
4.227 ms
1.391 ms
8.473 ms
4.379 ms
2.397 ms
2.415 ms
1.586 ms
1.452 ms
1.322 ms
1.428 ms
1.428 ms
2.847 ms
1.426 ms
1.355 ms
8.636 ms
4.294 ms
3.811 ms
1.350 ms
1.443 ms
1.331 ms
1.341 ms
1.348 ms
5.366 ms
4.823 ms
2.341 ms
1.373 ms
1.370 ms
1.317 ms
3.291 ms
1.315 ms
13.784 ms
1.302 ms
1.429 ms
5.592 ms
1.397 ms
1.660 ms
5.689 ms
1.413 ms
1.352 ms
1.363 ms
1.362 ms
6.264 ms
3.977 ms
2.323 ms
1.339 ms
5.612 ms
4.235 ms
7.046 ms
1.498 ms
1.304 ms
1.518 ms
1.323 ms
1.335 ms
1.315 ms
6.623 ms
1.332 ms
1.588 ms
1.424 ms
1.355 ms
1.376 ms
1.453 ms
1.338 ms
1.402 ms
1.331 ms
1.392 ms
1.325 ms
1.939 ms
2.444 ms
1.322 ms
1.677 ms
1.387 ms
1.370 ms
6.437 ms
1.368 ms
3.518 ms
1.433 ms
1.758 ms
1.332 ms
1.307 ms
1.337 ms
1.411 ms
1.426 ms
1.340 ms
1.400 ms
1.311 ms
1.370 ms
1.382 ms
1.307 ms
1.439 ms
1.301 ms
1.346 ms
1.350 ms
1.344 ms
1.373 ms
1.306 ms
1.456 ms
1.460 ms
1.397 ms
1.422 ms
1.360 ms
1.385 ms
1.402 ms
1.374 ms
1.335 ms
1.358 ms
1.331 ms
1.393 ms
1.309 ms
1.390 ms
1.365 ms
1.365 ms
1.497 ms
1.327 ms
1.313 ms
1.351 ms
1.450 ms
2.549 ms
1.281 ms
1.352 ms
1.351 ms
6.564 ms
1.368 ms
1.343 ms
1.336 ms
9.251 ms
1.346 ms
1.382 ms
1.332 ms
1.377 ms
1.478 ms
1.313 ms
1.329 ms
1.622 ms
1.315 ms
1.356 ms
1.322 ms
1.421 ms
1.345 ms
1.583 ms
1.413 ms
1.317 ms
1.308 ms
1.540 ms
1.370 ms
1.392 ms
1.325 ms
1.378 ms
1.314 ms
1.355 ms
1.325 ms
1.435 ms
1.335 ms
1.344 ms
1.340 ms
3.572 ms
1.332 ms
1.325 ms
1.500 ms
1.358 ms
1.344 ms
4.185 ms
1.557 ms
1.330 ms
1.387 ms
1.414 ms
1.400 ms
1.311 ms
1.359 ms
1.333 ms
1.408 ms
1.316 ms
3.632 ms
1.938 ms
1.347 ms
1.297 ms
1.351 ms
1.344 ms
1.517 ms
1.356 ms
1.331 ms
1.343 ms
1.390 ms
1.484 ms
1.358 ms
1.337 ms
1.294 ms
1.339 ms
1.345 ms
1.319 ms
1.770 ms
1.296 ms
1.298 ms
1.375 ms
1.333 ms
1.339 ms
1.328 ms
1.377 ms
1.354 ms
1.343 ms
1.329 ms
1.345 ms
1.507 ms
1.447 ms
1.375 ms
1.442 ms
1.372 ms
1.358 ms
1.320 ms
1.301 ms
1.307 ms
1.386 ms
2.647 ms
1.457 ms
1.325 ms
1.376 ms
1.322 ms
1.332 ms
1.352 ms
1.294 ms
1.413 ms
1.479 ms
Network path / recorded observations

Network path observations.

17 observations

Choose a measurement type to see the closest of the available sampled observations to the selected chart point. Each probe is separate and may follow a different path.

Before

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.129 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.174 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop9.100 msReply RTT ASN unavailable
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.135 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.181 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop21.054 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.073 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination0.901 msReply RTT AS13335 · Cloudflare
Before

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.122 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.177 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop8.289 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.149 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.196 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.709 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.120 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination2.178 msReply RTT AS13335 · Cloudflare
Before

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.262 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.255 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.284 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop2.325 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.525 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.128 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.293 msReply RTT AS8002 · Stealth
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.138 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.190 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop3.394 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.348 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.403 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.122 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.244 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop33.296 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hop1.493 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop2.172 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.389 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.139 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.186 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop13.109 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop4.465 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.258 msReply RTT AS13335 · Cloudflare
During

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.563 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.207 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.105 msReply RTT AS13335 · Cloudflare
During

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.473 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.289 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop37.299 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hop1.639 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop2.422 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Probe hopNo replyNo comparable RTT ASN unavailable
  7. TTL 7 Destination1.772 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.141 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.252 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.579 msReply RTT AS13335 · Cloudflare
After

TCP traceroute / 443 · Recorded address 2

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.220 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.170 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop0.806 msReply RTT AS13335 · Cloudflare
  4. TTL 4 Probe hop15.858 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop1.624 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.044 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 2

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.133 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.670 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop22.732 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.191 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.269 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.395 msReply RTT ASN unavailable
  4. TTL 4 Probe hop1.411 msReply RTT AS13335 · Cloudflare
  5. TTL 5 Probe hop1.646 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination0.851 msReply RTT AS13335 · Cloudflare
After

ICMP traceroute · Recorded address 1

Recorded trace · destination not reached by this trace


to UTC

  1. TTL 1 Probe hop0.269 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.184 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop14.450 msReply RTT ASN unavailable
After

TCP traceroute / 443 · Recorded address 1

Recorded trace · destination replied


to UTC

  1. TTL 1 Probe hop0.147 msReply RTT AS8002 · Stealth
  2. TTL 2 Probe hop0.186 msReply RTT AS8002 · Stealth
  3. TTL 3 Probe hop1.243 msReply RTT ASN unavailable
  4. TTL 4 Probe hopNo replyNo comparable RTT ASN unavailable
  5. TTL 5 Probe hop1.680 msReply RTT AS13335 · Cloudflare
  6. TTL 6 Destination1.427 msReply RTT AS13335 · Cloudflare

RTT is the round trip to each responding hop, not the delay of the preceding link. TTL order may omit routers and does not prove direct links. A missing reply is not proof of packet loss. ASN labels are dated address associations; they do not establish the forwarding path or the operator at the event time. Up to three recorded traces per method and period are selected across the archive. Where no individual trace is available, a comparison may summarize a window; MTR observations aggregate multiple probes. Address aliases are independent across measurement types.

Same method / same recorded address

Compare the traces.

Compare one selected trace from before the event with one during it. Values are individual replies, not a statistical comparison.

ICMP traceroute · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.129 msAS8002 · Stealth0.563 msAS8002 · StealthSame responder
20.174 msAS8002 · Stealth0.207 msAS8002 · StealthSame responder
39.100 ms1.105 msAS13335 · CloudflareSame responder
ICMP traceroute · Recorded address 2

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.122 msAS8002 · Stealth0.128 msAS8002 · StealthSame responder
20.177 msAS8002 · Stealth0.293 msAS8002 · StealthSame responder
38.289 msAS13335 · CloudflareNot recordedNot established
TCP traceroute / 443 · Recorded address 1

Before: UTC · During: UTC

Same destination address and probe method. A matching TTL does not establish the same forwarding path.
TTLBefore RTTDuring RTTResponding address
10.149 msAS8002 · Stealth0.138 msAS8002 · StealthSame responder
20.196 msAS8002 · Stealth0.190 msAS8002 · StealthSame responder
31.709 msAS13335 · Cloudflare3.394 msSame responder
4No replyNo replyNot established
52.120 msAS13335 · Cloudflare1.348 msAS13335 · CloudflareDifferent responders
6No reply1.403 msAS13335 · CloudflareNot established
72.178 msAS13335 · CloudflareNot recordedNot established

RTTs include the return path. Different responders, missing replies and independent probe flows limit what a change can localize. MTR snapshots remain separate in the replay.

Event timeline.

Latency anomaly. Times shown are UTC.

Event opened
UTC
Last event observation
UTC
Recovery observed
UTC

Recorded recovery describes this event, not current reachability.

02 / Assessment

Assessment.

Investigation stateRecovered
Path assessment

Elevated latency observed.

Endpoint round-trip time was elevated during part of the event. The available path observations do not identify where the increase occurred.

What we observed

  • A sustained increase in measured round trip time was observed.
  • Stored traces were compared using matching source and destination addresses.
  • Repeated replies from a comparable hop showed increased round trip time.

Recording counts describe collected measurements, not application availability or packet loss. Exact packet paths and the return path remain unverified.

Measurements at a glance.

Baseline RTT1.358 msUsual round trip before the event
Last recorded event RTT1.402 msLast successful event recording
Event p95 RTT7.579 ms95th percentile of successful recordings
Change from baseline+0.044 msLast event RTT compared with the baseline
Successful / expected recordings140 / 140Recorded endpoint samples during the event

RTT is round trip time; p95 describes the upper range of successful recordings. Endpoint measurements differ from individual trace replies and are not an uptime percentage.

Record sources and timestamps
Analysis updated
UTC
Public snapshot published
UTC
Public observation window
– UTC

Event and source timestamps are UTC. The latency chart uses its selected timezone. Page refresh times are separate from measurement timestamps.

03 / Method and limits

Method and limits.

How to interpret the sampled path and its limits

Corroborating the path.

Comparisons use matching destinations before and during the event. ICMP traces use echo probes. TCP/443 traces send SYN probes to port 443 only; they do not complete a TCP handshake or perform a TLS or HTTP exchange.

The Stealth gNMI interface verifies the active route-table choice and maps its primary next hop to the Stealth logical interface, configured attachment and, where applicable, the known LAG member set at each snapshot. Stable Stealth replies, together with mapped interface evidence showing no corresponding errors or sustained capacity pressure, reduce the likelihood of a Stealth forwarding issue. An 80% load warning warrants investigation; it does not prove congestion. Lower interval averages can conceal bursts or queue pressure. Optical readings assess the physical link, not congestion.

What remains unseen.

Probe reply handling can differ from packet forwarding. A slow intermediate reply that disappears at later hops does not locate a forwarding problem. A destination that does not respond to a trace is not thereby shown to be unavailable.

Stealth does not use ECMP in this monitored scope. The route snapshot does not identify which physical LAG member carried a particular independent probe, attest line-card forwarding at that packet instant, prove the exact application packet flow or observe the return path. RTT includes the return path, which may differ. No remote reverse traceroute is available. Tunnels and silent routers may hide parts of the path. Address registration alone does not establish the forwarding operator or responsibility.

References: ICMP reply handling / RFC 1812 · Round trip measurements / RFC 2681 · Full methodology