BYOKchat
BYOKchat Blog

Measuring Time to First Token Correctly

Learn how to measure AI time to first token without confusing DNS, TLS, request upload, provider queueing, first protocol event, reasoning, and first visible answer.

· 5 min read

On this page
  1. Start with a precise definition
  2. First byte is not first token
  3. Reasoning models make the definition more important
  4. Request start should be close to actual network dispatch
  5. Context construction can be a real latency source
  6. DNS and TLS can dominate cold requests
  7. Connection pooling changes measurements
  8. Proxies move the observation point
  9. Provider-reported latency is a different metric
  10. Clock choice matters
  11. Do not time from UI rendering completion if you mean network TTFT
  12. First token can be an empty or non-visible delta
  13. Tool-first responses complicate TTFT
  14. Distinguish queueing from generation speed
  15. Use distributions, not one average
  16. Segment comparisons carefully
  17. Prompt size can affect prefill latency
  18. Prompt caching can change TTFT
  19. Avoid fake precision
  20. Cancellation and failures should not be mixed into successful TTFT blindly
  21. Timeouts need separate instrumentation
  22. A practical metric schema
  23. Privacy-preserving analytics are enough
  24. Benchmarking checklist
  25. Where BYOKchat fits
  26. Further reading

“Time to first token” sounds like one number.

In practice, several different clocks can hide inside it:

request starts
→ DNS
→ TCP/TLS
→ request upload
→ provider edge
→ provider queue
→ model starts
→ first protocol event
→ first reasoning event
→ first visible answer text

If two clients measure different boundaries, their TTFT values are not comparable even when they call the same model.

The useful rule is:

Define exactly what starts the clock and exactly what event stops it. Keep transport, provider, and user-visible latency separate.

Start with a precise definition

A common client-side definition is:

TTFT_visible = timestamp(first visible answer delta)
             - timestamp(request dispatch)

That is useful because it approximates what the user feels after tapping Send.

But it includes more than model inference.

It can include:

  • local request construction;
  • connection establishment;
  • network latency;
  • proxy time;
  • provider queueing;
  • provider preprocessing;
  • model reasoning before visible output.

So label it honestly.

First byte is not first token

The client may receive HTTP response headers or a keepalive before any model content.

Track separately:

TTFB = first response byte - request start
TTFE = first semantic stream event - request start
TTFT = first visible text delta - request start

These can differ substantially.

For example:

request start       0 ms
headers           300 ms
keepalive         350 ms
reasoning event  1200 ms
visible text     2500 ms

Reporting 350 ms as “TTFT” would be misleading.

Reasoning models make the definition more important

A provider may stream reasoning summaries before visible answer text.

You might track:

time_to_first_event
time_to_first_reasoning
time_to_first_visible_answer

For product UX, first visible answer is usually the most intuitive latency metric unless the reasoning UI itself is intentionally user-visible.

See How Streaming Reasoning Differs From Streaming Answers.

Request start should be close to actual network dispatch

If you start timing when the user taps Send but then spend 500 ms:

  • loading attachments;
  • counting tokens;
  • building context;
  • serializing a giant request;

then the number measures send-to-visible latency, not just provider response latency.

That can still be useful.

Just separate stages:

user_send_at
request_build_started_at
request_dispatched_at
headers_received_at
first_event_at
first_visible_delta_at
completed_at

Then you can answer where time was spent.

Context construction can be a real latency source

A local-first client may do substantial work before the request:

load chat history
resolve project instructions
select attachments
retrieve relevant files
summarize old context
estimate tokens
truncate context
serialize request

If users report “the model feels slow,” provider TTFT may be fine while local preparation is the bottleneck.

Measure it.

DNS and TLS can dominate cold requests

A cold connection may require:

DNS lookup
TCP handshake
TLS handshake
HTTP negotiation

A warm connection may reuse existing infrastructure and skip much of that work.

This means latency distributions often have distinct cold/warm populations.

Do not compare one warm request to another client’s cold request and call the provider faster.

See the planned article on connection reuse later in the roadmap.

Connection pooling changes measurements

If your networking stack reuses HTTP/2 or HTTP/3 connections, later requests can have much lower setup latency.

Track enough context to interpret results:

provider host
network type
connection reuse if observable
request sequence/session context

Avoid invasive fingerprinting; you only need diagnostic metadata relevant to your own network stack.

Proxies move the observation point

In a proxied architecture:

client → your backend → provider

The client measures:

client-to-backend + backend-to-provider + provider work + return path

Your backend can also measure:

backend request received
provider request started
provider first event
backend first event forwarded

Comparing both sides helps detect:

  • proxy queueing;
  • response buffering;
  • slow serialization;
  • network distance.

Provider-reported latency is a different metric

Some providers expose server timing or request metadata.

If available, treat it as provider-side observation, not a replacement for client latency.

Useful comparison:

client TTFT: 1800 ms
provider processing to first output: 900 ms

The missing 900 ms may include network/proxy/client stages.

Clock choice matters

Use a monotonic clock for duration measurement.

Wall-clock time can jump because of:

  • system time adjustments;
  • NTP corrections;
  • timezone changes.

For example:

start = monotonicNow()
...
elapsed = monotonicNow() - start

Store wall-clock timestamps separately if you need chronological logs.

Do not time from UI rendering completion if you mean network TTFT

The first text delta can arrive at time t1, but the UI may render it at t2.

Track both if performance matters:

first_visible_delta_received_at
first_visible_delta_rendered_at

A heavy Markdown renderer can make user-visible latency worse even when provider TTFT is good.

See How to Render Markdown While AI Is Still Streaming.

First token can be an empty or non-visible delta

Provider protocols may emit:

  • item-start events;
  • role metadata;
  • empty text parts;
  • reasoning metadata;
  • content block starts.

Your adapter should stop the visible TTFT clock only when meaningful visible text arrives.

Do not let wire protocol details pollute the metric.

Tool-first responses complicate TTFT

The model may decide to call a tool before writing user-facing text.

Sequence:

request
→ tool call at 700 ms
→ tool execution 3 s
→ final text at 4.2 s

Possible metrics:

time to first model action = 700 ms
time to first visible text = 4.2 s

Both are valuable.

If you report only first visible text, the model can look slow even though it quickly made the correct tool decision.

If you report only first model event, you hide the user’s wait.

Distinguish queueing from generation speed

A model can have slow TTFT but fast generation once it starts.

Another can start instantly but generate slowly.

Track:

TTFT
stream duration after first visible token
visible output tokens
average generation rate

See AI Generation Speed Explained: Tokens per Second.

Use distributions, not one average

Latency is usually skewed.

Prefer reporting:

median (p50)
p90
p95
p99 when sample size supports it

One slow outage can distort the arithmetic mean.

For a small local analytics screen, median plus a high percentile is often more understandable.

Segment comparisons carefully

TTFT depends on:

  • provider;
  • model;
  • region/network path;
  • prompt size;
  • cached input;
  • reasoning setting;
  • tool configuration;
  • cold/warm connection;
  • current provider load.

Do not compare models without controlling or labeling these variables.

Prompt size can affect prefill latency

A request with 500 input tokens and one with 100,000 input tokens are not equivalent.

Longer input may require more preprocessing/prefill before output can begin.

If you want meaningful client analytics, store sanitized usage metadata such as:

input token count
cached token count if provider reports it
model
reasoning setting
TTFT

No prompt text is required.

Prompt caching can change TTFT

Provider-side prompt caching may reduce work for repeated prefixes, depending on provider/model semantics.

So two requests with equal input token counts may have different latency because one reuses cached computation.

If the provider reports cache usage, keep it alongside the latency measurement.

See AI Prompt Caching Explained.

Avoid fake precision

A UI showing:

TTFT: 1.237483 s

suggests accuracy that the system does not have.

For user-facing analytics, values such as:

1.24 s

or rounded milliseconds are enough.

More precision can remain in raw diagnostic telemetry if useful.

Cancellation and failures should not be mixed into successful TTFT blindly

A request that fails before first text has no successful visible TTFT.

Store:

TTFT = null
failure_stage = before_first_visible_output

Do not assign its duration as TTFT.

Likewise, user cancellation before output should be a separate outcome.

Timeouts need separate instrumentation

If you enforce:

first-event timeout = 30 s

record whether the timeout fired before:

  • headers;
  • first protocol event;
  • first visible text.

Otherwise you cannot tune it intelligently.

A practical metric schema

type GenerationTiming = {
  startedAt: number;
  requestDispatchedAt?: number;
  headersAt?: number;
  firstEventAt?: number;
  firstReasoningAt?: number;
  firstVisibleTextAt?: number;
  completedAt?: number;
};

Derived values:

request_build_ms
header_latency_ms
first_event_ms
first_reasoning_ms
visible_ttft_ms
total_duration_ms

Privacy-preserving analytics are enough

You can analyze streaming performance without storing:

  • prompt text;
  • response text;
  • reasoning text;
  • attachment contents;
  • credentials.

Useful sanitized fields:

provider
model
input/output token counts
reasoning setting
TTFT
duration
finish category
network class at coarse level if appropriate

This is enough for most performance diagnostics.

Benchmarking checklist

When comparing models/providers:

  • use the same client/network path;
  • use similar prompt sizes;
  • separate cold and warm runs;
  • keep reasoning settings comparable;
  • record tool use separately;
  • use enough samples;
  • compare distributions;
  • note prompt-cache effects;
  • report first visible text consistently;
  • avoid claiming universal provider performance from one location.

Where BYOKchat fits

A multi-provider client can instrument timing at the shared generation layer while adapters declare what counts as reasoning, visible text, tool action, and completion. This makes cross-provider analytics useful without pretending their wire events are identical.

The most important metric is the one whose boundary is documented.

Further reading

Keep reading