Observability for Streaming AI Requests

A streaming AI request is not one latency number.

Between the user pressing Send and the final response, a client can move through:

local queue
network setup
provider processing
first stream event
first visible token
continuous generation
tool call
user approval
tool execution
follow-up model round
semantic completion
local persistence

If observability records only:

duration = 18.4s
success = true

then almost every useful debugging question remains unanswered.

A good telemetry model preserves the shape of the lifecycle while excluding prompts, responses, reasoning, credentials, tool arguments/results, and private endpoint details.

Start with two identities

Every operation should have a logical request ID:

logical_request_id

Each network attempt should have a separate attempt ID:

attempt_id

Why both?

user action
  └─ logical request 123
       ├─ attempt A -> 503
       ├─ attempt B -> stream interrupted
       └─ attempt C -> completed

The product should count one user action. The reliability layer should see three provider attempts.

Preserve provider request IDs when safe

Many APIs return a request/trace ID useful for support and provider diagnostics.

Store it as sanitized metadata when available:

provider_request_id

Do not confuse it with your own logical request ID.

Your ID survives retries across providers; the provider’s ID belongs to one specific attempt.

Define lifecycle timestamps explicitly

Useful timestamps include:

user_action_at
queued_at
attempt_started_at
request_sent_at
headers_received_at
first_provider_event_at
first_visible_token_at
last_event_at
semantic_completion_at
persisted_at

Not every provider exposes every boundary, so fields can be optional.

The important part is that names have stable definitions.

TTFT needs a precise start point

There are at least two useful TTFT-like metrics:

attempt TTFT = first visible token - attempt start
user TTFT = first visible token - user action

The first helps compare provider/model behavior.

The second reflects real product responsiveness, including local queue time.

Do not mix them under one metric name.

See Measuring Time to First Token Correctly.

First event is not always first token

Streaming APIs may emit:

response created
message start
role metadata
reasoning event
tool event
usage metadata
heartbeat

before visible text.

Track both if useful:

time_to_first_event
time_to_first_visible_token

This separates network/provider responsiveness from answer rendering.

Record queue wait independently

If the user waits 5 seconds before a network worker is available, provider TTFT should not become 5 seconds worse.

Track:

queue_wait_ms = attempt_started - queued

and include it in user-perceived latency separately.

See How to Build an AI Request Queue.

Duration needs phase breakdown

A useful request duration can include:

queue wait
connect/first-response
pre-token processing
stream generation
tool wait
tool execution
retry delay
persistence

A 60-second total can be healthy if 50 seconds were an intentional long-running tool and unhealthy if 50 seconds were a dead socket.

Phase timing makes the difference visible.

Tokens per second needs a stable interval

A common generation-speed metric is:

$$\text{generation speed} = \frac{\text{output tokens}}{\text{generation interval}}$$

But decide what the interval means.

For example:

first generated token -> final generated token

is different from:

request start -> completion

The second includes TTFT and should not be labeled pure decoding/generation speed.

See AI Generation Speed Explained: Tokens per Second.

Token counts need provenance

Possible sources include:

provider-reported usage
client tokenizer estimate
stream event accounting
unknown

Record the source:

input_tokens = 1234
input_token_source = provider

Do not silently compare provider-reported exact usage with client-side estimates as if they were identical.

Cache usage should be a separate dimension

If the provider exposes cached-input accounting, keep it distinct from ordinary input tokens.

Useful fields might be:

input_tokens
cached_input_tokens
cache_write_tokens
output_tokens

Provider semantics differ, so normalize carefully and preserve unknown fields only where they are useful.

See AI Prompt Caching Explained.

Finish state is more useful than success/failure

Streaming needs richer outcomes:

completed
cancelled_by_user
failed_before_output
failed_after_partial_output
context_limit
rate_limited
policy_stopped
tool_failed
deadline_exceeded
interrupted_unknown

A partial answer followed by disconnect should not be indistinguishable from a request that failed before any output.

Preserve semantic completion boundaries

An adapter should record whether it observed the provider’s expected completion event/finish state.

This helps distinguish:

connection closed after valid completion

from:

connection disappeared unexpectedly

The text may look complete in both cases.

Error taxonomy belongs in telemetry

Record stable categories such as:

transport
http_auth
http_permission
rate_limit
provider_transient
model_capability
context_limit
policy
tool
protocol_parse
local_app
cancelled
unknown

Keep raw provider error codes separately when safe.

See How to Classify AI API Errors.

Retry events need their own trace

For each retry, record:

attempt number
prior failure category
retry delay
server retry-not-before if any
remaining logical deadline
final attempt outcome

Then you can answer:

How often do retries recover requests?
How much latency do retries add?
Are retries multiplying because SDK and app both retry?

without inspecting user content.

Tool calls create nested spans

A tool-enabled trace can look like:

Record spans such as:

model_round_1
tool_wait_for_approval
tool_execution
model_round_2

This reveals whether slow “AI requests” are actually waiting on tools.

Never log tool arguments/results by default

Tool payloads can contain:

email addresses
calendar details
file contents
SQL results
private URLs
credentials

Useful telemetry is usually structural:

tool name or safe tool ID
duration
approved/denied
success/failure category
bytes/result-size bucket if safe
round number

See Privacy-Preserving Analytics for AI Apps.

Approval time is a human phase

If a request waits 20 seconds for the user to approve a tool, do not blame the provider.

Track:

approval_wait_ms

separately from model and tool execution.

This also prevents circuit breakers from interpreting long human waits as service timeouts.

Reasoning and answer streams may need separate counters

Some APIs expose reasoning-related events separately from answer text.

You can track structural timing:

reasoning_started_at
answer_started_at

without storing reasoning content.

Be careful with token accounting because providers differ in what they expose and bill.

Do not infer hidden reasoning details that the API does not report.

Structured output needs validation telemetry

A request can complete at the transport level but fail application validation.

Track:

provider_completed = true
structured_validation = failed

instead of turning it into a generic HTTP failure.

Useful subcategories include:

invalid JSON
schema mismatch
truncated output
unsupported schema behavior

See Structured AI Output Explained.

Background jobs need durable trace identity

A provider job can outlive the app process.

Persist:

logical request ID
provider job ID
created_at
last_status_at
completion source = poll/webhook/reconcile
final state

On relaunch, continue the same logical trace rather than starting a new anonymous operation.

See How Long-Running AI Tasks Work.

App lifecycle is part of the trace

On mobile, record lifecycle transitions relevant to active work:

entered_background
suspended/connection interrupted if observable
resumed
reconciled

Avoid treating these as provider failures.

See How to Handle App Backgrounding During AI Generation.

Network-path changes explain latency shifts

Privacy-safe network metadata can be coarse:

Wi-Fi / cellular / other
expensive/constrained flags where platform exposes them
local/private endpoint vs public provider

Do not collect SSIDs, IP addresses, hostnames, or private endpoint URLs unless there is an explicit need and privacy policy.

The goal is to explain broad performance changes without fingerprinting the user.

Connection reuse should be observable indirectly

You may not always have direct socket-level visibility, but timing can show connection setup cost.

If your networking stack exposes connection metrics, track coarse phases such as:

DNS
connect
TLS
request
response

Connection reuse often reduces the first three phases on later requests.

See How Connection Reuse Affects AI Latency.

Separate local model metrics from cloud metrics

Local inference can expose different useful signals:

model load time
prompt processing time
queue depth
tokens per second
memory pressure indicator

A local request with no network setup should not be forced into cloud-centric fields.

Use a shared high-level schema plus operation-specific optional dimensions.

Metrics need bounded cardinality

Dangerous dimensions include:

full prompt
full URL
conversation title
file path
arbitrary error message
raw tool arguments

They create privacy risk and explode metric cardinality.

Prefer stable enums/IDs:

provider_kind
model_id where safe/bounded
error_category
operation_class
finish_state

For custom providers, consider a generic custom category rather than uploading the user’s hostname.

Request IDs are still identifiers

Provider request IDs and local logical IDs can be useful for debugging, but decide retention and scope carefully.

They should not become durable cross-product user identifiers.

For telemetry exported off-device, consider whether hashed/rotating IDs or aggregate metrics satisfy the use case.

Local-only observability can be much richer

If analytics stays entirely on-device, the app can retain detailed diagnostic traces without transmitting them.

Even then, avoid secrets and unnecessary conversation content because local logs may appear in backups, diagnostics, or support exports.

Good minimization is useful regardless of destination.

One trace can drive both analytics and developer diagnostics

A structured event model can power:

per-request diagnostics
aggregate latency charts
error breakdowns
provider health
retry analysis
tool performance

without separate ad hoc log formats.

The UI can render sanitized trace records when developer mode is enabled.

A possible trace schema

interface AIRequestTrace {
  logicalRequestId: string;
  providerKind: string;
  modelId?: string;
  operationClass: string;
  queueWaitMs?: number;
  attemptCount: number;
  ttftMs?: number;
  durationMs: number;
  outputTokens?: number;
  generationTokensPerSecond?: number;
  toolCallCount: number;
  retryCount: number;
  finishState: string;
  errorCategory?: string;
}

Keep detailed attempt/tool spans in child records when necessary instead of flattening everything into one huge event.

Aggregate by the question you are answering

Examples:

Responsiveness

user-action-to-first-token
queue wait
TTFT

Generation performance

output tokens
stream generation interval
tokens/sec

Reliability

semantic completion rate
partial interruption rate
transient failure rate
retry recovery rate

Tools

tool success rate
tool duration
approval wait
round count

Do not combine them into one “performance score.”

Traces should survive partial failures

Write important lifecycle state incrementally or hold enough in memory to finalize a trace after error/cancellation.

A trace that exists only on successful completion systematically hides the failures you most need to debug.

Persist finalization idempotently

If the app crashes between provider completion and trace finalization, relaunch reconciliation should not create duplicate analytics events.

Use stable logical IDs and upsert/finalize semantics where practical.

This is especially important for durable background jobs.

Clock choice matters

Use monotonic clocks for in-process elapsed durations.

Use wall-clock timestamps for persisted ordering and cross-process recovery.

Do not derive precise elapsed performance solely from wall-clock differences if the device clock can change during the request.

Test instrumentation, not only request logic

Use deterministic fake streams:

normal completion
slow first token
long steady stream
partial stream reset
429 then retry success
user cancellation
tool approval delay
tool timeout
structured-output validation failure
background/relaunch recovery

Assert the generated trace fields.

Instrumentation bugs can silently ruin months of metrics even while the app behaves correctly.

An observability checklist

• Logical request IDs are separate from provider attempts.
• Provider request IDs are preserved only as safe metadata.
• Queue wait, TTFT, streaming, tools, and total duration are distinct.
• First event and first visible token are not conflated.
• Token counts record their source.
• Finish states preserve cancellation and partial failures.
• Errors use normalized categories.
• Retry delays and recovery are visible.
• Tool/approval phases are nested rather than blamed on the provider.
• Background operations keep durable trace identity.
• Metrics use bounded, privacy-safe dimensions.
• Prompts, responses, reasoning, credentials, tool payloads, private URLs, and attachment contents are excluded.
• Failed requests still produce traces.

Where BYOKchat fits

A local-first BYOK client can make observability useful without sending conversation content to an analytics service. Sanitized request records can capture tokens, estimated cost, request count, TTFT, duration, generation speed, tool calls, retries, and reliability while keeping prompts, responses, reasoning, credentials, tool arguments/results, attachment contents, URLs, and hostnames out of telemetry.

That same trace model can power developer diagnostics and provider-health signals while preserving the privacy expectations of direct-to-provider chat.

Further reading

• Privacy-Preserving Analytics for AI Apps
• Measuring Time to First Token Correctly
• AI Generation Speed Explained
• How to Classify AI API Errors
• Health Scoring AI Providers Without Fake Precision