How Streaming Tool Calls Work

Streaming tool calls are easy to misunderstand because the model can emit a tool request incrementally.

A client might receive something that looks like:

{"city":"San

then:

 Francisco","unit":"c

then:

elsius"}

If the tool is executed before the provider says that call is complete, the client can run with malformed or incomplete arguments.

The central rule is:

Streamed tool-call fragments are transport/protocol state, not executable arguments. Execute only after the provider-defined call boundary is complete and the full argument object has been validated.

Tool calling has two different streams

Conceptually, a generation can contain:

model output stream
  ├─ visible text deltas
  ├─ reasoning events
  ├─ tool-call metadata
  ├─ tool-call argument deltas
  └─ completion events

Then the application may start another phase:

complete tool call
→ validate
→ authorize
→ execute tool
→ send result to model
→ stream continuation

So “streaming tool calls” is not the same as “stream tool execution.”

A tool request is structured model output

A complete call usually needs at least:

tool identity
call identity
arguments

Depending on provider, the wire representation may include:

index
item ID
call ID
function name
arguments as JSON text
arguments as structured object
finish/completion boundary

Do not assume every provider uses the same fields.

Arguments often arrive as text fragments

One common protocol pattern streams JSON argument text incrementally.

For example:

fragment 1: {"query":"
fragment 2: latest MCP spec
fragment 3: ","limit":5}

The client should assemble:

{
  "query": "latest MCP spec",
  "limit": 5
}

before parsing/validation.

Trying to JSON.parse() each fragment individually is wrong.

Do not assume chunk boundaries match JSON token boundaries

A provider can split anywhere:

{"li
mit":
5}

or inside escaped strings:

{"text":"line 1\nli
ne 2"}

The safe approach is append-only buffering keyed by the provider’s call/item identity.

type PendingToolCall = {
  callId: string;
  name?: string;
  argumentText: string;
  complete: boolean;
};

Parallel tool calls require separate buffers

A model can request multiple independent tools in one model turn.

Conceptually:

call A: get_weather(city="Paris")
call B: get_exchange_rate(pair="EURUSD")

Their argument fragments may be interleaved in the stream.

Do not keep one global toolArgumentsBuffer.

Use a map keyed by stable call identity:

pendingCalls[callId].argumentText += delta;

If the protocol only gives an index initially, the adapter may need to map index → call record until a stable ID is available.

Parallel does not mean dependent

If the model emits two tool calls in the same turn, they should generally be treated as peer requests from that model decision.

Do not infer:

call B depends on result of call A

If B truly needs A’s result, the model normally needs another model/tool round:

model → call A
app → result A
model → call B using result A

This distinction matters for schedulers and execution ordering.

Provider completion boundaries are authoritative

A tempting optimization is:

argument buffer currently parses as valid JSON
→ execute now

That is unsafe.

The current buffer may be valid JSON but still not final.

Example:

{"query":"cats"}

is valid, but the model may still append data if the protocol has not declared the item finished.

Use the provider’s event semantics to know when the call is complete.

Valid JSON is not valid tool input

After the argument stream finishes:

parse the completed representation;
validate it against the tool schema;
apply application/business validation;
apply authorization/policy;
only then execute.

Pipeline:

Each boundary catches a different class of failure.

Tool schemas are not authorization policies

A schema can validate:

{
  "path": "/tmp/report.txt"
}

It does not answer:

Is this model allowed to read that file?

Likewise:

{
  "recipient": "ceo@example.com",
  "body": "..."
}

can be structurally valid while still requiring user approval.

Keep authorization outside model-generated arguments.

Streamed names need validation too

Some protocols may expose the function/tool name as metadata before or alongside arguments.

Do not execute an arbitrary model-supplied string by dynamic reflection.

Resolve against the registered tool catalog:

requested name
→ exact known tool definition
→ schema + policy + executor

If the tool is unknown or disabled, return a controlled tool error/result according to the provider loop rather than calling arbitrary code.

Tool-call IDs are correlation state

The model/provider may require the client to return a tool result associated with one call ID.

Persist that relationship through execution:

provider call ID
↕
local execution ID
↕
returned result

Do not generate a new call ID when sending the result unless the provider protocol requires it.

Your local execution ID can be separate

A robust app may create its own durable ID:

provider_call_id = call_abc
local_execution_id = exec_123

The local ID can survive:

retries;
app relaunch;
provider changes;
internal tool infrastructure.

This is useful for idempotency and auditability.

Side effects make duplicate stream handling important

Imagine the client receives the same completed tool call twice because of:

provider retry;
reconnect replay;
client parser bug;
duplicated event delivery.

A read-only search might be harmless to repeat.

A send_email call is not.

Keep duplicate detection and idempotency at the execution layer.

See Designing Reliable AI Retries.

A tool call can complete while the overall response does not

There are several levels of completion:

argument field complete
tool call item complete
model response requires tool result
model response turn complete
final assistant answer complete

Do not collapse them.

A client can finish assembling a call while the model is still emitting another parallel call.

Depending on provider semantics, execution may wait until the current model output boundary or may be allowed once each call is finalized. Follow the provider protocol rather than guessing.

Streaming multiple calls needs deterministic ordering

The model may produce:

call index 0
call index 1
call index 0 more args
call index 1 more args

Preserve provider order/identity so you can reconstruct the complete model output accurately.

Do not sort calls alphabetically by tool name or local execution completion time before sending results unless the API permits it.

Tool execution order is a separate decision

If two finalized calls are independent and read-only, the application may execute them concurrently.

If they mutate shared state, the application may need serialization even if the model requested them in parallel.

Examples:

get_weather + get_stock_price → parallel likely fine
rename_file + delete_file → ordering may matter
create_record + update_same_record → dependency unclear

Execution policy belongs to the host.

Cancellation can happen during argument streaming

If the user stops generation before a call is finalized:

partial call ≠ executable call

Discard the incomplete call buffer from execution state, though you may keep diagnostic metadata.

Never “salvage” a half-streamed tool call by guessing missing arguments.

Cancellation can happen after execution starts

This is harder.

Suppose:

model call complete
→ user approval
→ tool starts upload
→ user taps Stop

You need separate cancellation capabilities:

cancel model generation
cancel local tool if tool supports it

A tool may be non-cancellable once the side effect commits.

The UI should not claim rollback unless the tool actually supports rollback.

Interrupted streams require durable tool state

If the stream breaks after a side-effecting call was executed but before model continuation, persist:

call ID
arguments hash/reference
approval decision
execution status
result
side-effect identity/idempotency key

Then recovery can send the existing result instead of executing the tool again.

See How to Resume or Recover an Interrupted AI Generation.

Tool results can be much larger than calls

A tool may return:

many search results;
file contents;
logs;
structured records;
binary references.

Do not assume the provider can accept unlimited result size.

The host may need:

result truncation
summarization
resource references
pagination
context budgeting

Preserve enough provenance that the user can inspect what happened.

Tool errors are normal model-loop inputs

A tool can fail because:

arguments are invalid;
authorization denied;
remote service unavailable;
file missing;
user cancels approval;
execution timeout;
internal application error.

Not every failure should terminate the whole chat.

Depending on provider protocol and product policy, the app can send a structured tool error back to the model so it can adapt.

Do not send raw stack traces, credentials, or sensitive internal details.

Approval UI should use the complete call

Never ask the user to approve a half-streamed action.

Wait until you know:

which tool
complete arguments
which account/server
what side effect it represents

Then show a meaningful approval card.

For example:

Send email
To: alice@example.com
Subject: Project update

is much safer than:

Tool wants permission
Arguments: {"to":"ali...

Separate model intent from host authority

The model decides:

I would like to call tool X with arguments Y

The host decides:

Is X enabled?
Are Y valid?
Does policy allow this?
Does the user need to approve?
Which account/environment should execute it?

That separation remains true even when the call arrives via a streaming protocol.

A provider-neutral internal representation

After native events are fully decoded, a useful neutral call might look like:

type ToolCall = {
  providerCallId: string;
  toolName: string;
  arguments: unknown;
  providerMetadata?: unknown;
};

The core app should not need to know that one provider originally streamed arguments as JSON text while another emitted structured content blocks.

Keep opaque metadata only where continuation requires it.

A streaming adapter state machine

The important transition is Collecting → Ready: it should be driven by provider semantics, not by “JSON happens to parse.”

Tests that catch real bugs

Test:

arguments split one byte at a time;
Unicode split inside an argument string;
escaped JSON sequences split across chunks;
two parallel calls interleaved;
three calls sharing the same tool name but distinct IDs;
valid JSON before provider completion;
invalid JSON at completion;
unknown tool name;
schema mismatch;
disabled tool;
approval denied;
duplicate completed-call event;
disconnect during argument collection;
disconnect after execution before result is sent;
cancellation during execution;
tool returns very large result;
provider requires result ordering/correlation.

Do not build provider logic into the tool executor

The executor should ideally receive a complete validated host-level call.

Bad boundary:

tool executor parses Anthropic/OpenAI/Gemini stream events

Better:

provider adapter
→ complete neutral tool call
→ policy/approval
→ tool executor
→ neutral tool result
→ provider adapter serializes result

This keeps both sides testable.

Where BYOKchat fits

A multi-provider BYOK client can normalize tool-call completion into one host workflow while still preserving provider-native IDs and continuation metadata. That allows the same approval policy and MCP/tool execution layer to work across providers without pretending their streaming event formats are identical.

The important invariant is simple: no tool runs from an incomplete stream fragment.