How AI Streaming Works: Tokens, SSE, Tool Calls, and Reasoning Events

When an AI chat app feels fast, the model usually has not finished the answer. The client is rendering a stream of partial events while generation continues on the server.

That sounds like “print tokens as they arrive,” but production streaming is more complicated. A stream can contain text, reasoning metadata, tool-call arguments, usage updates, completion markers, and errors. Some of those values are complete objects; others arrive as fragments that are invalid on their own.

A reliable client therefore needs to separate three things:

1. transport framing — how bytes arrive over the network;
2. provider events — what those bytes mean according to one API;
3. application events — the stable concepts your UI and persistence layer understand.

If those layers are mixed together, small provider changes turn into renderer bugs.

The request is ordinary HTTP until it is not

Most text streaming APIs begin as a normal HTTP request:

POST /some-generation-endpoint
Authorization: Bearer ...
Content-Type: application/json

The client sends the whole request body up front. The important difference is in the response: instead of waiting for one final JSON document, the server keeps the connection open and sends incremental data.

For many AI APIs this is done with Server-Sent Events, or SSE.

At the wire level an SSE response resembles:

event: some_event
data: {"type":"some_event","delta":"Hel"}

event: some_event
data: {"type":"some_event","delta":"lo"}

Each event is separated by a blank line. The data: field contains the payload. Some APIs use named event: lines; others mostly send data: records with the event type inside JSON.

SSE is only framing. It does not define what an AI delta means.

Streaming has layers

A good mental model is:

Each layer should have one job.

The network layer should not know about Markdown. The Markdown renderer should not know what an Anthropic content_block_delta is. The persistence layer should not append raw SSE strings to a message record.

That separation is what makes multi-provider streaming manageable.

A token is not necessarily a network chunk

People often say that models “stream tokens.” That is directionally useful but technically imprecise.

The model generates tokens internally. The serving system may buffer several tokens before sending them. HTTP/TCP may split or combine bytes differently again. Your JavaScript, Swift, or Python networking API may expose chunks at still another boundary.

So this assumption is unsafe:

one network chunk = one token = one visible character fragment

A network chunk might contain:

• half of one UTF-8 character;
• one complete SSE event;
• several SSE events;
• part of a JSON payload;
• several text tokens grouped together.

Your transport parser must reconstruct protocol frames first. Only then should the provider parser inspect JSON.

UTF-8 boundaries matter

Suppose a stream contains an emoji. UTF-8 encodes that character using multiple bytes. A network read can split those bytes.

If you decode each raw byte chunk independently, you can produce replacement characters or corrupt text.

Use an incremental UTF-8 decoder that preserves incomplete sequences between reads. Most mature HTTP/streaming libraries already provide this abstraction, but hand-written parsers sometimes get it wrong.

The same principle applies at every layer:

Buffer until you have a complete unit for the layer you are parsing.

For bytes, that unit may be valid text. For SSE, it is an event frame. For JSON, it is normally the complete data: payload—unless the provider deliberately defines a field as a partial JSON string.

Providers expose different semantic event models

The transport may be similar while the event vocabulary differs significantly.

OpenAI Chat Completions

A streamed Chat Completions response is traditionally represented as completion chunks. Each chunk may contain a delta for content, role information, tool calls, finish state, or usage depending on configuration and API behavior.

The application generally accumulates deltas until it can construct the final assistant message.

OpenAI Responses

Responses uses a more explicitly typed event model. Text deltas are only one kind of event. The stream can describe output-item lifecycle and other response state, which is useful when a turn contains more than plain text.

That is one reason an application should normalize the provider stream instead of letting OpenAI event names leak into UI code. See OpenAI Responses API vs Chat Completions.

Anthropic Messages

Anthropic’s current streaming protocol uses named SSE events. A message begins with message_start, content blocks have start/delta/stop lifecycles, top-level message updates arrive through message_delta, and the stream ends with message_stop.

A tool-use content block can emit input_json_delta events whose partial_json value is not necessarily valid JSON yet. It must be accumulated for that content block before parsing the completed arguments.

Gemini

Gemini’s APIs expose more than one streaming surface. streamGenerateContent returns streamed GenerateContentResponse chunks over SSE, while the newer Interactions API exposes typed streaming events such as step deltas.

Again, the transport being SSE does not make the application event model identical.

Normalize events before touching the UI

A provider-neutral client benefits from a small event vocabulary such as:

type GenerationEvent =
  | { type: "started" }
  | { type: "textDelta"; text: string }
  | { type: "reasoningDelta"; text: string }
  | { type: "toolCallStarted"; id: string; name: string }
  | { type: "toolCallArgumentsDelta"; id: string; fragment: string }
  | { type: "toolCallFinished"; id: string }
  | { type: "usage"; inputTokens?: number; outputTokens?: number }
  | { type: "completed"; finishReason?: string }
  | { type: "failed"; error: GenerationError };

This is not a universal standard. It is an application contract.

The provider adapter maps wire-level events into that contract:

Now the UI can render textDelta without caring which provider produced it.

Treat a generation as a state machine

Appending strings works for a toy demo. A real generation should have explicit state.

For example:

idle
  ↓
connecting
  ↓
streaming
  ├── text
  ├── reasoning
  ├── tool call
  └── usage updates
  ↓
completed

And failures can happen from several states:

connecting ──→ failed
streaming  ──→ failed
streaming  ──→ cancelled

A reducer can own this state:

type GenerationState = {
  status: "connecting" | "streaming" | "completed" | "failed" | "cancelled";
  text: string;
  reasoning: string;
  toolCalls: Map<string, ToolCallState>;
  usage?: Usage;
};

This gives cancellation, retries, tool execution, and interrupted-stream recovery somewhere coherent to live.

Tool-call arguments are the classic streaming trap

Suppose a model wants to call:

{
  "city": "Ho Chi Minh City",
  "units": "metric"
}

A provider might stream the arguments as fragments:

{"city":"Ho
 Chi Minh
 City","units":"met
ric"}

Or as fragments aligned differently. Until the tool call is complete, this string may not parse as JSON.

Do not execute a tool because a partial buffer happens to parse early. A provider’s tool/content completion boundary is necessary but not sufficient: an enclosing response can still be incomplete or truncated, such as when a token limit is reached. After the boundary, verify the provider’s stop/completion state, parse the accumulated arguments, validate them against the tool schema and application policy, and only then execute the tool.

A safe lifecycle is:

This distinction matters with providers that can close a tool content block while the overall generation is incomplete. A structural end marker tells you where the argument fragments stop; it does not prove the model finished them successfully.

Reasoning events should not be treated as ordinary answer text

Providers differ in how reasoning is represented. It may be:

• visible text intended for the user;
• a summary;
• an opaque encrypted item that must only be round-tripped;
• metadata that should not be rendered at all.

Do not create a single text buffer and dump every delta into it.

At minimum, keep separate channels for:

final answer text
reasoning / thinking presentation
opaque provider continuation state
tool calls

This prevents protocol state from accidentally becoming visible UI content and makes it possible to collapse or omit reasoning without breaking continuation.

Streaming Markdown is a parser problem, not just a paint problem

A partial Markdown document is often syntactically incomplete.

Imagine these successive buffers:

Here is **important

then:

Here is **important text** and a code block:

```swift
let value =
```

then later more prose or another fence arrives.

If your renderer parses the whole document on every tiny delta, several problems appear:

• repeated parsing becomes expensive;
• incomplete syntax can cause visual flipping;
• code blocks may appear and disappear while fences are incomplete;
• selection and scroll positions can jump;
• syntax highlighting may run far too often.

A strong streaming renderer normally separates generation cadence from render cadence.

For example, network events may arrive dozens of times per second while the UI batches updates to one render per animation frame or to a small time window.

network deltas:  ██████████████████████████
UI commits:       █   █   █   █   █   █

The exact strategy depends on platform, but the principle is stable: do not tie expensive Markdown work directly to every packet.

Preserve raw text independently from rendered output

Your durable assistant message should usually store source text, not generated HTML or attributed UI fragments.

Why?

• the renderer will evolve;
• themes change;
• Markdown bugs get fixed;
• exports need source semantics;
• other platforms may render differently.

A useful split is:

stream accumulator → canonical Markdown source
                         ↓
                    renderer cache
                         ↓
                         UI

When generation finishes, you can perform a full final render and replace any streaming approximation.

Cancellation has two jobs

When the user taps Stop, the client should:

1. stop consuming and rendering the current stream;
2. cancel the underlying request when the transport/API supports it.

Doing only the first leaves network and server work running unnecessarily. Doing only the second without updating local state can leave the UI stuck in a generating state.

Cancellation should be modeled as a terminal generation outcome distinct from failure:

completed ≠ cancelled ≠ failed

That distinction matters for “Continue”, regenerate, analytics, and recovery after relaunch.

An HTTP 200 does not mean the generation succeeded

With non-streaming JSON, applications often treat the HTTP status as the main success boundary.

Streaming changes that. The server can accept the request, return 200 OK, begin sending data, and then fail halfway through generation.

Possible causes include:

• provider-side generation errors;
• upstream gateway failures;
• tool failures;
• connection loss;
• malformed events;
• application cancellation;
• device network transitions.

So success is not:

HTTP status == 200

It is closer to:

valid stream start
+ valid event sequence
+ provider completion signal
+ final state committed

A client should retain the partial text if that is useful, but mark the turn as interrupted rather than silently treating it as complete.

Retry is dangerous after partial output

If the request fails before any output arrives, a retry can often be straightforward.

After partial output arrives, retry semantics become ambiguous. A new request may generate different text. If tools were already executed, replaying the turn can duplicate side effects.

Classify the generation before retrying:

State at failure	Typical strategy
Connection failed before response	Retry may be safe
Stream opened, no semantic output	Usually retryable with care
Partial answer text received	Prefer explicit regenerate/continue UX
Read-only tool completed	Retry may still change model trajectory
Mutating tool executed	Do not automatically replay without idempotency protection

This is one reason provider fallback is not simply a networking retry. See How to Build Reliable AI Provider Fallback and Model Routing.

Usage may arrive late

Token accounting is another subtlety. Providers may report usage near the end of the stream rather than alongside each text fragment. Some counters are cumulative; some describe the final response only.

Do not estimate final billing by counting visible text characters.

Keep usage as its own stream state and commit the authoritative provider values when available:

type Usage = {
  inputTokens?: number;
  outputTokens?: number;
  cachedInputTokens?: number;
  reasoningTokens?: number;
};

Not every provider exposes every field, so optionality is part of the model.

Backpressure and slow consumers

A fast model can produce data faster than an expensive UI pipeline wants to process it.

This is especially noticeable when every delta triggers:

• Markdown parsing;
• syntax highlighting;
• database writes;
• layout recalculation;
• scroll animations.

The answer is not to deliberately slow the network parser. Keep transport consumption lightweight and batch expensive downstream work.

A practical pipeline is:

stream read
  ↓
parse event
  ↓
append to in-memory generation state
  ↓
throttled UI update
  ↓
periodic / terminal persistence

Writing the entire conversation database on every two-character delta is avoidable work.

Stream parsing must be forward-compatible

Provider event vocabularies evolve. Anthropic explicitly documents that clients should tolerate unknown event types, and the same defensive principle is useful generally.

A robust adapter should distinguish:

• unknown but ignorable events;
• malformed events;
• known error events;
• protocol violations that make continuation unsafe.

Logging an unknown event for diagnostics is better than crashing the chat renderer because a provider introduced one new metadata event.

How to test a streaming implementation

Do not test only with a fast happy-path model response.

A meaningful test matrix includes:

Transport fragmentation

Feed the parser one byte at a time, arbitrary chunk sizes, and multiple events in one chunk. The semantic result should be identical.

Unicode

Split multi-byte UTF-8 characters across input chunks.

Markdown boundaries

Split at **, backticks, code fences, table rows, links, and math delimiters.

Tool arguments

Split JSON at every possible byte boundary and ensure execution happens only after the provider marks the call/content boundary complete, the enclosing generation is not truncated or incomplete, and the accumulated arguments validate.

Cancellation

Cancel before first output, during text, during reasoning, and while waiting for a tool.

Mid-stream failure

Inject a disconnect after partial text and verify the UI marks the turn interrupted.

Unknown events

Insert a syntactically valid event the adapter does not recognize and confirm the intended forward-compatible behavior.

Slow rendering

Artificially delay UI work and verify the stream parser remains correct.

Measure the right latency

Users experience several different latencies:

request start
   ↓
connection / queue
   ↓
first semantic output        ← TTFT-ish user experience
   ↓
continuous generation
   ↓
last output
   ↓
final accounting / completion

Time to first token, time to first visible text, time to first useful content, and total generation duration are not always identical.

A reasoning model may emit non-visible reasoning state before answer text. A tool-using model may produce a tool call quickly but not show final text until after execution.

Record timestamps for meaningful lifecycle events instead of trying to infer everything from one duration number.

A compact implementation checklist

Before calling a streaming client reliable, verify that it:

• incrementally decodes UTF-8;
• parses complete SSE frames independently of network chunking;
• maps provider events into a neutral event model;
• keeps text, reasoning, tools, and opaque continuation state separate;
• accumulates tool JSON until an explicit completion boundary, then verifies the enclosing generation is complete enough to use it;
• validates tool arguments before execution;
• batches expensive Markdown/UI updates;
• handles cancellation at both network and application levels;
• distinguishes completed, failed, cancelled, and interrupted turns;
• persists authoritative usage when the provider reports it;
• tolerates expected unknown events;
• tests arbitrary fragmentation and mid-stream failure.

Where BYOKchat fits

A multi-provider client cannot afford to make the chat renderer understand every provider’s streaming protocol. BYOKchat’s useful architectural boundary is the provider adapter: provider-native streams can stay native at the network edge, then become common chat/tool events before they reach shared conversation and rendering code.

That preserves provider capabilities without coupling the UI to one SSE vocabulary.

Further reading

• Anthropic streaming Messages documentation
• Gemini streaming interactions
• OpenAI Responses API reference
• OpenAI Responses API vs Chat Completions
• How a BYOK AI Chat Request Works