Why AI Streams Break in the Middle

The hardest AI streaming failures happen after the request has already succeeded.

The client receives an HTTP 200 response, renders several paragraphs, and then the stream stops.

At that point, “the request failed” is too vague to be useful.

A mid-stream failure can come from:

the user’s network;
an intermediary proxy or VPN;
your own backend;
the AI provider;
the model execution itself;
malformed stream framing;
your parser;
app suspension;
user cancellation;
a tool call that never completes;
a timeout policy in any layer.

The useful question is:

Which layer stopped making progress, and what evidence tells us whether the generation itself still exists?

A stream has multiple failure boundaries

A typical path might be:

Every hop can fail independently.

The visible symptom can be identical:

last token arrived 8 seconds ago
connection closed

So diagnosis needs more than a generic network error message.

Failure before first byte is different from failure after deltas

Separate at least these stages:

1. DNS / connect
2. TLS
3. request upload
4. response headers
5. first stream event
6. active streaming
7. protocol completion
8. local persistence/render completion

A failure at stage 2 means the provider probably never saw the request.

A failure at stage 6 means the provider definitely started responding, and may have continued running even after the client lost the connection.

That difference matters for retries and cost.

A 200 response is only transport setup success

This sequence is possible:

HTTP 200
→ text deltas
→ tool call
→ provider error event
→ EOF

The HTTP status says the stream was opened successfully. It does not guarantee the generation completed.

The adapter should wait for a provider-defined successful terminal condition such as:

explicit completed event;
final response object;
finish reason;
provider completion sentinel.

If the connection closes without that condition, treat the outcome as incomplete unless the API guarantees otherwise.

Network handoff can kill a healthy generation

Mobile users frequently move between:

Wi-Fi → cellular
cellular → Wi-Fi
VPN off → VPN on
one access point → another

A TCP or HTTP connection may not survive that transition.

The provider may still be generating normally on the server.

From the client’s perspective:

provider operation: maybe alive
network stream: dead

This is one reason durable provider response IDs or background-job APIs are valuable when available.

Cellular networks can be deceptively unstable

A connection can fail because of:

radio transitions;
NAT mapping changes;
carrier proxies;
weak signal;
temporary packet loss;
captive portal changes;
power-saving behavior.

Do not assume an apparently good speed test means a long-lived stream will be stable.

For testing, use network conditioning that introduces:

high latency;
intermittent drops;
packet loss;
bandwidth changes;
full disconnect/reconnect.

VPNs and security products add another intermediary

Corporate VPNs and filtering proxies can:

terminate TLS;
inspect long-lived responses;
enforce request duration limits;
buffer stream chunks;
close idle-looking connections;
rewrite headers;
reject WebSocket upgrades;
behave differently on HTTP/2 versus HTTP/1.1.

If failures happen only on one enterprise network, do not immediately blame the AI provider.

Capture enough metadata to compare:

network type
proxy/VPN presence if observable without invasive fingerprinting
HTTP protocol version
request duration
last event timestamp
bytes received
provider request ID

Reverse proxies can silently buffer output

Your backend may receive provider deltas immediately while the user receives them in bursts.

That can happen if a reverse proxy buffers response data instead of forwarding it incrementally.

Symptoms include:

backend logs: delta every 50 ms
client: nothing for 8 seconds, then 100 KB arrives

This is a delivery problem, not slow model generation.

When diagnosing TTFT or stuttering, measure timestamps at multiple layers if you operate a proxy.

Idle timeout is not the same as overall timeout

A proxy might allow a request to run for ten minutes as long as bytes keep flowing, but close it after 60 seconds of silence.

Another system may enforce a hard 120-second total request duration regardless of activity.

Distinguish:

connect timeout
response-header timeout
first-event timeout
idle-stream timeout
overall generation timeout

Reasoning-heavy models can create long silent periods depending on how the provider exposes reasoning events. An idle timeout that was fine for fast text-only models can suddenly become too aggressive.

Keepalives can prevent false idle detection

Some streaming protocols send comments or heartbeat events that carry no model content.

For SSE, a server can send a comment line such as:

: keepalive

The client should not render it, but the bytes can keep intermediaries from treating the connection as idle.

Whether you can control this depends on which layer owns the stream.

If you proxy a provider stream, think carefully before inventing heartbeats: they keep the client connection active, but they do not prove the upstream provider generation is healthy.

Provider-side failures can arrive after many tokens

A model request can fail late because of:

internal inference errors;
quota/accounting changes;
tool infrastructure failure;
policy checks;
overloaded runtime;
provider service disruption;
backend worker crash.

A robust adapter should surface provider-native error information without leaking raw sensitive payloads into analytics.

Useful stored fields include:

provider
model
HTTP status if any
provider error category/code
provider request ID
stream stage
received output length
elapsed time

Malformed stream data is its own failure class

The connection can stay open while the payload becomes invalid.

Examples:

invalid JSON inside data:
unknown event shape
truncated UTF-8
line exceeds parser limit
unexpected event ordering
missing tool-call identifier

Do not report these as network failures.

They indicate:

provider protocol drift;
incompatible proxy behavior;
OpenAI-compatible endpoint mismatch;
parser bug;
corrupted data.

See How to Parse Server-Sent Events Correctly.

Compatible APIs often fail at semantic boundaries

An endpoint may claim OpenAI compatibility and successfully emit ordinary text deltas, but fail when:

reasoning events appear;
parallel tool calls stream;
usage is included at the end;
structured output is enabled;
finish reasons differ;
error objects use another schema.

The stream transport can be perfectly healthy while your adapter cannot decode the events.

Classify that as protocol incompatibility, not connectivity failure.

Tool calling creates long non-text phases

A generation may look like it “stopped” because the model switched from visible text to tool execution.

Example:

text delta
text delta
model emits tool call
client executes tool for 20 seconds
client sends tool result
model resumes

If the UI only watches visible text timestamps, it may falsely declare the stream stalled.

Track the generation phase:

awaiting model
streaming text
streaming reasoning
assembling tool call
executing tool
waiting for continuation
completed

Timeouts should respect the current phase.

Tool execution can outlive the model stream

Some provider protocols end one stream when tool calls are emitted. The application executes tools, then starts a new model request with results.

From the user’s perspective this is one “generation,” but technically it may be multiple request streams.

Your generation state machine should therefore sit above individual HTTP requests.

Otherwise a successful end of the first request might be mistaken for a successful final answer.

App backgrounding can interrupt streams

On mobile, the application can lose foreground execution while a response is streaming.

Depending on OS policy and networking APIs:

the process may continue briefly;
the socket may remain open temporarily;
the app may be suspended;
the process may be terminated later;
background transfer APIs may not support arbitrary streaming request semantics.

Do not promise that a long foreground stream can always survive app backgrounding.

Persist enough state so the app can recover when relaunched.

See How to Handle App Backgrounding During AI Generation later in this roadmap.

User cancellation is not a failure

If the user taps Stop, the UI should not show:

Generation failed: network connection lost

Cancellation is an intentional terminal state.

Model it separately:

type GenerationEnd =
  | { kind: "completed" }
  | { kind: "cancelled_by_user" }
  | { kind: "failed"; error: GenerationError }
  | { kind: "interrupted"; recoverable: boolean };

That distinction also improves analytics.

Client-side renderer crashes can look like stream failures

The network may continue receiving events while UI code crashes or blocks.

Examples:

Markdown parser hits pathological input;
syntax highlighter performs expensive work repeatedly;
state mutation triggers excessive re-rendering;
main thread is blocked;
database write deadlocks or stalls;
observation/state framework throws.

Measure network receipt separately from UI presentation.

If your logs show new deltas arriving after the user says “it froze,” the bottleneck is likely downstream of the socket.

Partial output must be preserved deliberately

When a stream breaks after substantial text, you have three choices:

discard it;
keep it as partial output;
try to recover/continue.

Discarding can be frustrating and destroys diagnostic context.

A better local record might store:

assistant turn
status = interrupted
visible_text = partial output
provider_response_id = optional
last_event_at
failure_category

The UI can show an interrupted-state affordance rather than pretending the text is a completed answer.

Retrying the whole request can duplicate side effects

If the generation was pure text, regeneration is usually conceptually safe, though it may cost again and produce different output.

If tools were involved, blindly retrying can be dangerous.

Suppose:

model requested create_invoice
client executed it
stream broke before assistant confirmation

Retrying the whole turn can cause another tool call and duplicate invoice creation unless tool execution is idempotent.

See Designing Reliable AI Retries and Idempotency for AI Tool Execution later in the roadmap.

Recovery depends on what is durable

Possible recovery strategies include:

Regenerate from last safe conversation state

Best when:

no side-effecting tools executed;
provider operation cannot be recovered;
partial answer is not authoritative.

Continue from provider response state

Possible when the provider exposes durable continuation objects.

Poll a background operation

Best when the provider supports long-running/background responses.

Reconnect to your own event log

Possible when your backend persists provider events and can replay them by sequence/operation ID.

Keep partial output and ask user

Often the safest fallback when semantics are unclear.

A useful error taxonomy

Avoid one giant streamError enum.

A practical taxonomy can include:

transport.connectivity
transport.timeout
transport.proxy_closed
transport.tls
protocol.invalid_sse
protocol.invalid_json
protocol.unexpected_event
protocol.unexpected_eof
provider.rate_limit
provider.internal
provider.policy
provider.auth
application.cancelled
application.backgrounded
application.tool_failed
application.parser_bug

Not every product needs this exact hierarchy, but separate retryability from user-facing wording.

Retryability is not binary by error code alone

For example, a network disconnect may be retryable in principle but unsafe after a side-effecting tool executed.

So retry policy needs context:

type RetryContext = {
  failure: FailureCategory;
  receivedAnyOutput: boolean;
  toolSideEffectsOccurred: boolean;
  providerOperationId?: string;
  cancelledByUser: boolean;
};

Then decide whether to:

retry automatically;
offer regenerate;
reconcile provider state;
show partial result;
require user confirmation.

Instrument stream progress, not just start/end

Useful timestamps include:

request_started_at
headers_received_at
first_event_at
first_visible_text_at
last_event_at
completion_event_at
request_closed_at

Also track counts:

bytes received
semantic event count
text delta count
tool event count

This lets you distinguish:

provider slow to start
vs
provider started then stalled
vs
network delivered but UI stalled

Heartbeat-style stall detection

For an active stream, you can detect “no progress” using the last meaningful transport or protocol event timestamp.

But be cautious:

long reasoning phases may be quiet;
tool execution can take time;
provider heartbeats may count as transport activity but not model progress;
client backgrounding changes timer reliability.

Use stall detection to improve diagnostics and UX, not to assume the generation is dead with absolute certainty.

Build a deterministic failure simulator

Production-only stream bugs are painful.

Your provider simulator should be able to reproduce:

close before headers
close after 1 event
close after N tokens
pause for 60 seconds
emit invalid SSE
emit invalid JSON
duplicate event
emit provider error event
finish without terminal event
stream tool call then disconnect
finish tool then disconnect before continuation

Then test the exact local state left behind after every scenario.

A recovery-oriented state machine

A useful abstraction:

This is more useful than a single isLoading Boolean.

Debugging checklist

When a stream breaks, ask:

Did response headers arrive?
Did any semantic event arrive?
Was there an explicit provider error event?
Did a valid terminal event arrive?
When was the last byte/event?
Did the user cancel?
Was the app backgrounded?
Was a tool running?
Did a proxy enforce an idle/total timeout?
Does the provider expose a request/response ID?
Can provider state be queried after disconnect?
Did the parser reject malformed data?
Did UI rendering stop while network events continued?
Could retry duplicate a side effect?

Where BYOKchat fits

A local-first multi-provider client can retain partial assistant turns and provider metadata even when the network stream disappears. Each adapter can classify provider-specific completion/error events while the shared generation state machine handles cancellation, tool phases, interruption, and recovery consistently.

That turns a broken stream from “everything disappeared” into an explicit, inspectable state.