AI Generation Speed Explained: Tokens per Second

Tokens per second is one of the most useful AI performance metrics—and one of the easiest to misuse.

At a high level:

generation speed = output tokens / generation time

But the result depends on:

which tokens you count;
where the timer starts;
whether reasoning tokens are included;
whether the provider reports token usage accurately;
whether you exclude time before the first output token;
whether tool execution pauses the model loop;
whether tokenizer differences make cross-model comparisons meaningful.

The key rule is:

Tokens per second measures output throughput during a defined interval. It is not the same thing as response latency, model quality, or total task speed.

TTFT and generation speed are different

Consider two models.

Model A:

TTFT = 0.4 s
generation = 20 tokens/s

Model B:

TTFT = 2.0 s
generation = 80 tokens/s

For a 20-token answer, A may feel faster.

For a 2,000-token answer, B may finish much sooner.

So track both:

latency to start
throughput after start

See Measuring Time to First Token Correctly.

The simplest formula

If you know:

first output timestamp
completion timestamp
output token count

then:

tokens_per_second = output_tokens / (completion_time - first_output_time)

Example:

output tokens = 800
first output = t + 1.2 s
completed = t + 11.2 s

Then:

800 / 10 = 80 tokens/s

This measures post-start output throughput.

Do not include TTFT unless you label it differently

If you instead calculate:

800 / 11.2 = 71.4 tokens/s

that includes startup latency.

That metric may still be useful, but it is closer to:

end-to-end effective output rate

not pure generation throughput.

Use names that make the boundary obvious.

Provider-reported output tokens are usually better than client estimation

If the provider reports final usage:

{
  "input_tokens": 1200,
  "output_tokens": 800
}

prefer that output-token count for provider billing/performance analytics.

Client-side tokenization can disagree because:

tokenizer versions differ;
chat formatting contributes hidden tokens;
reasoning accounting differs;
provider-specific special tokens are unknown;
compatible endpoints may use different tokenizers.

See Why Different AI Models Count Tokens Differently.

Character rate is not token rate

A tempting fallback is:

characters / 4 ≈ tokens

That can be acceptable for a rough UI estimate, but it is not robust enough for precise benchmarking.

Token density changes with:

language;
whitespace;
code;
punctuation;
tokenizer vocabulary;
Unicode text.

If usage is unavailable, label the result estimated.

Visible text tokens may not equal billed output tokens

Reasoning-capable models may perform or report reasoning work separately.

The provider’s output usage can include categories that are not directly visible in the rendered answer.

So distinguish:

visible text throughput
provider-billed output token throughput
reasoning-inclusive throughput if reported

Do not derive invisible reasoning speed from answer length.

Streaming event count is not token count

One stream event can contain:

part of a token;
one token;
many tokens;
metadata only.

So this is wrong:

number of delta events / seconds = tokens per second

Protocol batching is an implementation detail.

Use actual usage or tokenizer estimates.

Tool calls create gaps that distort throughput

Suppose:

model outputs 100 tokens
→ tool call
→ tool takes 10 seconds
→ model outputs 200 tokens

If you calculate:

300 tokens / total 15 seconds

then the tool’s runtime makes model throughput look slower.

That may be useful for user-visible task speed, but it is not pure model generation speed.

For diagnostics, track phases:

model streaming duration
tool duration
continuation streaming duration

Multi-round agent turns need more than one throughput number

A tool-using turn can involve multiple model requests:

request 1 → 80 output tokens → tool
request 2 → 400 output tokens → final answer

You can report:

request 1 TPS
request 2 TPS
aggregate model output TPS across active model intervals
end-to-end task duration

Do not compress everything into one number unless the UI explains what it means.

Average speed can hide burstiness

A stream may arrive:

0-2 s: 0 tokens
2-3 s: 150 tokens
3-4 s: 20 tokens
4-5 s: 120 tokens

Average throughput is useful, but it does not show stalling or burst delivery.

For deeper diagnostics, measure per-window throughput:

tokens in each 500 ms or 1 s interval

This can reveal:

provider batching;
proxy buffering;
renderer backpressure;
network stalls.

Use such data for debugging, not as flashy user-facing precision.

Proxy buffering can fake slow/fast bursts

Your backend may receive tokens smoothly while the client receives them in chunks.

Client-side throughput then measures delivery throughput, not necessarily provider model throughput.

If you operate a proxy, compare timestamps at both boundaries:

provider event received by backend
client event received

If they diverge, investigate buffering.

Rendering speed is another independent bottleneck

The client can receive 100 tokens/s but visually update at 10 Hz because it coalesces rendering.

That can be good.

Do not calculate token throughput from UI frame updates.

Measure stream receipt at the generation layer, then optimize rendering separately.

Completion timestamp must be defined consistently

Possible endpoints:

last text delta
provider completed event
network EOF
final usage event
UI finished rendering

For model output throughput, use the provider’s semantic completion boundary where possible.

If usage arrives slightly after the last text delta, the difference may be small but should be consistent.

A robust metric schema

type RequestPerformance = {
  requestStartedAt: number;
  firstVisibleOutputAt?: number;
  completedAt?: number;
  inputTokens?: number;
  outputTokens?: number;
  reasoningTokens?: number;
  providerReportedUsage: boolean;
};

Then derive:

TTFT
post_first_output_duration
output_tokens_per_second
end_to_end_tokens_per_second (optional)

Do not compare TPS across tokenizers as if tokens were equal units of text

One tokenizer may encode a sentence in 20 tokens while another uses 27.

So a model emitting:

80 tokens/s

is not necessarily producing twice as much human-readable text as one at:

40 tokens/s

Cross-provider TPS comparisons are approximate performance indicators, not a universal characters-per-second measure.

Consider complementary character/word throughput for UX research

If you need human-perceived text speed, you can also measure:

characters/s
words/s

from visible output.

These are easier to compare across tokenizers, but they also vary by language and content style.

Use them as complementary metrics, not replacements for provider token accounting.

Language affects apparent speed

The same token rate can produce different visible character rates in English, Vietnamese, Chinese, code, or mathematical notation.

So a user may perceive one answer as “typing faster” even when reported TPS is identical.

This is another reason not to overinterpret the number.

Reasoning effort can change both TTFT and throughput profile

Higher reasoning settings may:

delay visible output;
consume more hidden reasoning work;
change answer length;
change total task duration.

Do not compare low-effort and high-effort runs as pure model speed tests unless that difference is the point of the experiment.

See Reasoning Effort Explained.

Prompt size can indirectly affect generation

Large input primarily affects preprocessing/prefill and TTFT, but it can also change model behavior, output length, and cache use.

When benchmarking, record:

input tokens
cached input tokens if reported
output tokens
reasoning setting
model

This gives context to the observed throughput.

Model/provider load varies over time

Cloud model speed can change because of:

provider load;
capacity allocation;
routing;
regional infrastructure;
model revisions;
temporary incidents.

One measurement is not a durable benchmark.

Use distributions over time if you want reliability insights.

Local model TPS behaves differently

For local inference, throughput can depend heavily on:

device chip;
available memory;
model quantization;
context length;
thermal state;
concurrent workload;
inference engine;
batch/cache configuration.

Local TPS is still useful, but the environment must be part of the result.

Measure p50 and tail behavior

For many requests, store distributions such as:

TPS median
TPS p10/p90
TTFT p50/p95

A model that is usually 70 TPS but occasionally falls to 5 TPS may feel less reliable than one consistently around 50 TPS.

Do not hide variance behind a single average.

Avoid ranking models solely by speed

A useful model decision includes:

capability
quality for task
context support
reasoning/tool support
reliability
cost
TTFT
TPS
privacy/deployment constraints

Generation speed is one dimension.

A fast wrong answer is not a better outcome.

Example calculation

Suppose:

request dispatched:       0.000 s
first visible token:      1.500 s
provider completed:       9.500 s
provider output tokens:   640

Then:

TTFT = 1.5 s
active output duration = 8.0 s
TPS = 640 / 8.0 = 80 tokens/s

Optional end-to-end effective rate:

640 / 9.5 = 67.4 tokens/s

Label those metrics differently.

Failures and cancellations

If a response is interrupted after 300 tokens, you can still compute partial throughput for diagnostics.

But do not mix incomplete samples into successful performance distributions without a flag.

Store:

status = interrupted
partial_output_tokens = estimated/reported if available
partial_tps = ...

That lets you study whether stalls correlate with failures.

Privacy-preserving analytics

TPS requires no conversation content.

You only need metadata such as:

provider
model
request timestamps
usage counts
status
reasoning setting

This makes it a good local analytics feature for BYOK clients.

Benchmarking checklist

When comparing generation speed:

define TTFT separately;
use provider-reported output tokens when possible;
label estimated usage;
exclude tool execution from pure model TPS;
treat each model request separately in multi-round workflows;
record input size/reasoning settings;
compare distributions, not one run;
avoid cross-tokenizer overclaiming;
distinguish client delivery from provider generation if proxying;
keep incomplete/cancelled samples labeled.

Where BYOKchat fits

A multi-provider client can compute sanitized TTFT, duration, and TPS locally from provider usage plus semantic stream timing. Because the adapter knows which events are visible text, tool activity, reasoning, and completion, the shared analytics layer can avoid counting transport events as tokens or tool delays as model throughput.

That makes performance data useful without collecting prompts or responses.