Why Different AI Models Count Tokens Differently

The same text does not have one universal AI token count.

A prompt that is 4,800 tokens for one model might be 4,300 or 5,200 tokens for another because models can use different tokenizers, vocabularies, chat serialization, multimodal accounting, and provider-side protocol overhead.

That creates a practical rule for client developers:

Token count belongs to a specific model/API request, not to the text itself.

Text does not contain tokens inherently

The string:

Build a SwiftUI view that streams Markdown safely.

is a sequence of Unicode characters.

Tokens appear only after a tokenizer interprets that sequence:

text
  ↓
Tokenizer A → token IDs for Model A
Tokenizer B → token IDs for Model B
Tokenizer C → token IDs for Model C

Different tokenizers can choose different boundaries and therefore different counts.

Why model vocabularies differ

A tokenizer has a finite vocabulary of reusable pieces.

One vocabulary might contain a frequent sequence as one token:

SwiftUI

Another might split it:

Swift + UI

A third could divide it differently.

Vocabulary design depends on factors such as:

training corpus composition;
tokenizer algorithm;
vocabulary size;
supported languages;
code representation;
model generation;
special protocol tokens.

There is no requirement that providers use the same vocabulary.

Tokenizers are part of the model contract

Developers sometimes treat tokenization as a generic preprocessing detail.

For context management, it is closer to a model capability.

Your model profile may need information such as:

type TokenAccounting = {
  tokenizerFamily?: string;
  exactLocalCounterAvailable: boolean;
  contextLimit?: number;
  maxOutputTokens?: number;
};

Even then, a tokenizer family name is not necessarily enough to reproduce server-side accounting exactly.

Chat APIs add structure beyond raw text

Suppose the visible conversation is:

User: Hello
Assistant: Hi!
User: Summarize this file.

The provider may serialize that into model input using hidden or special delimiters representing:

role boundaries;
message starts/ends;
tool call blocks;
image references;
reasoning state;
system/developer instructions.

So this can be false:

requestTokens = tokenize(concatenateVisibleMessages)

An exact local tokenizer for raw strings may still differ from final provider usage.

Special tokens can change with API format

A model family may support multiple APIs or message formats.

The same semantic request could be represented as:

chat messages

or:

typed response/input items

or:

content parts with role metadata

Each representation can have different framing overhead.

That means even two endpoints using the same underlying model can report slightly different input usage for equivalent user content.

Multilingual text exposes tokenizer differences quickly

English-centric rules of thumb become unreliable across languages.

Consider:

Hello, how are you?
こんにちは
Xin chào, bạn khỏe không?
你好

Different tokenizer generations may represent these languages with very different efficiencies.

A vocabulary trained with better coverage of a language can often represent common sequences using fewer tokens than an older or less optimized vocabulary.

For a multilingual app, benchmark representative user text instead of extrapolating from English.

Unicode normalization can matter

Two strings can look identical while using different underlying Unicode sequences.

For example, accented characters can sometimes be represented as:

precomposed character

or:

base character + combining mark

A tokenizer can split those forms differently.

Applications should normally preserve user text rather than rewriting it solely to chase token counts, but normalization differences are useful to remember when debugging surprising estimates.

Source code has its own distribution

Programming languages contain patterns unlike prose:

private func makeRequest<T: Decodable>(
    _ route: Route<T>
) async throws -> T

Tokenizers differ in how efficiently they encode:

indentation;
punctuation;
camelCase identifiers;
snake_case identifiers;
operators;
generic syntax;
repeated code keywords.

A model optimized for coding workloads may have vocabulary patterns that make common code sequences cheaper to represent, but this is model-specific.

Do not assume prose ratios apply to repository-scale code.

JSON can be deceptively token-heavy

Structured data often repeats keys and punctuation:

{
  "id": "item_123",
  "status": "active",
  "metadata": {
    "owner_id": "acct_456"
  }
}

The token cost depends on how the tokenizer handles quotes, braces, common keys, underscores, and opaque IDs.

Large tool schemas and API payloads can therefore consume significant context even when their byte size seems modest.

Random strings are difficult to compress into vocabulary pieces

Natural language contains common recurring sequences.

Random data does not.

Examples:

UUIDs
SHA hashes
API keys
base64
signed URLs
opaque provider IDs

These often split into many tokens.

It is another reason secrets and binary-like blobs should not be placed in prompts. Besides being unsafe, they can be token-inefficient.

Tokenizer upgrades can change counts

Providers release new model generations.

A new model may use:

a different tokenizer;
a larger vocabulary;
improved multilingual coverage;
different special tokens;
different chat framing.

If your app stores a historical field like:

tokenCount = 18342

without recording which model/tokenizer produced it, the number has limited meaning later.

Prefer metadata such as:

{
  estimatedTokens: 18342,
  modelID: "provider/model-version",
  estimatorVersion: "..."
}

for estimates that need auditing.

Provider-reported usage can differ from local estimation

Even if you have the published tokenizer, the provider can add information your local counter does not know about.

Examples include:

internal formatting;
tool serialization;
image tokenization;
cached prefix accounting;
provider-managed conversation state;
reasoning continuation items.

Therefore this should be expected:

local estimate: 9,840
provider report: 9,917

The goal is not always zero difference. The goal is a conservative estimate that prevents context failures and a reported value that supports accounting after execution.

Different providers expose usage differently

One provider may return:

{
  "input_tokens": 1000,
  "output_tokens": 200
}

Another may return nested categories for:

cached input;
reasoning/thinking;
audio;
image input;
accepted/rejected generated tokens;
other model-specific usage.

A multi-provider app should normalize common fields while preserving native detail.

For example:

type NormalizedUsage = {
  inputTokens?: number;
  outputTokens?: number;
  cachedInputTokens?: number;
  reasoningTokens?: number;
  native: ProviderUsagePayload;
};

Do not force unsupported fields to zero. unknown and 0 are different facts.

Context capacity can also have provider-specific semantics

A model advertised with a large context window does not necessarily allow every combination of:

maximum input
+
maximum output
+
reasoning
+
all tool state

at their individual advertised maxima simultaneously.

Some APIs treat generated tokens as part of the same context budget. Others expose additional constraints or model-specific output ceilings.

Tokenization differences are only one part of context budgeting.

See Context Window vs Output Limit vs Reasoning Tokens.

Why a universal token counter is attractive

A multi-provider app wants one function:

countTokens(text)

It is simple to use for:

UI estimates;
preflight warnings;
context trimming;
cost previews.

But if the app supports arbitrary models, that function cannot be universally exact.

A better interface makes uncertainty explicit:

enum TokenEstimateQuality {
  case exactModelTokenizer
  case compatibleTokenizer
  case heuristic
  case unavailable
}

struct TokenEstimate {
  let count: Int
  let quality: TokenEstimateQuality
}

The caller can then decide how large a safety margin is needed.

Use model-specific counters when available

Best case:

selected model
→ known tokenizer
→ exact/official local implementation
→ request-aware overhead model

This can give high-confidence preflight counts.

But do not make the entire app depend on having a tokenizer for every provider. Custom OpenAI-compatible endpoints may expose unknown models that the client has never seen.

Graceful fallback matters.

Use provider count endpoints when they exist

Some providers expose a token-counting endpoint or SDK capability.

This can be more accurate than a local heuristic because the provider understands the selected model and request representation.

Tradeoffs include:

an extra network round trip;
potential rate limits;
privacy implications of sending content for counting;
availability/offline constraints;
latency before every request.

Do not automatically preflight every user message over the network unless the benefit justifies it.

Heuristics are still useful

A heuristic estimator can answer:

This request is probably ~20k tokens, not ~120k.

That is valuable.

It becomes dangerous only when the UI presents the estimate as exact or the context manager operates with no margin.

A robust sequence is:

Safety margins should depend on estimate quality

If you know the exact tokenizer and request format, you can use a smaller margin.

If you are estimating an unknown third-party model from characters, use a larger one.

For example, the policy can be conceptual rather than hard-coded globally:

exact model counter → modest protocol/output reserve
compatible counter  → larger reserve
heuristic            → conservative reserve + earlier compaction

The numbers should be measured against real provider reports.

Measure estimator error by model

Store content-free metrics such as:

estimated input tokens
reported input tokens
absolute error
percentage error
model ID
estimator version

Then compute:

Error\% = \frac{|reported - estimated|}{reported} \times 100

This reveals where your estimator is trustworthy without recording the user’s prompt.

Do not compare model efficiency using token counts alone

Suppose Model A uses fewer tokens than Model B for the same English text.

That does not automatically mean A is:

cheaper;
faster;
more intelligent;
better at long context.

Pricing rates, model architecture, throughput, context capacity, and quality all matter.

Tokens are model-native units, not a universal unit of intelligence or computation.

Token counts are not directly comparable across models

One generated token from Model A is not necessarily the same amount of text as one generated token from Model B.

Therefore:

Model A: 100 tokens/s
Model B: 100 tokens/s

is not a perfectly normalized character or word throughput comparison.

Tokens-per-second remains useful within a model/provider context, but cross-tokenizer comparisons need caution.

We cover this in AI Generation Speed Explained: Tokens per Second in the streaming phase.

Context compaction should use the selected model

Imagine a conversation has enough history for:

Model A → 80% context utilization
Model B → 96%

Switching the chat from A to B should trigger a new context calculation.

Do not cache one conversation token count and reuse it across models.

A good context manager accepts model capabilities as an input:

buildContext(
  conversation: Conversation,
  model: ModelCapabilities
) -> PreparedContext

See How to Switch AI Providers Mid-Conversation.

Prompt caching does not remove tokenizer differences

A cached prefix can reduce cost or latency, but the prefix still belongs to the selected model’s tokenization and caching scheme.

If you change models, do not assume:

same text → same cache token boundaries → same cache behavior

Caching is provider/model-specific.

See AI Prompt Caching Explained.

Practical testing corpus

To validate token estimation, test more than English prose.

Include:

short English
long English
Vietnamese
Chinese/Japanese/Korean
Arabic or other scripts relevant to users
emoji
Swift/Python/JavaScript code
JSON
Markdown tables
URLs
UUIDs/hashes
large tool schemas
mixed text + code

Record estimated and reported usage per model.

This catches the cases where a comfortable heuristic suddenly becomes unsafe.

Common design mistakes

Storing one token count on each message

A message does not have one model-independent token count.

Assuming OpenAI-compatible means tokenizer-compatible

An OpenAI-compatible endpoint can use completely different models and tokenizers.

Hard-coding a tokenizer by provider name

Providers can serve multiple model generations with different tokenization.

Using exact-looking UI for heuristic values

Display ~12k or “estimated” when that is what you know.

Ignoring provider-reported usage

The server’s final accounting is valuable feedback for improving estimates.

A multi-provider token-accounting checklist

Bind token estimates to a selected model.
Record estimator quality/version if estimates are persisted.
Support unknown/custom models gracefully.
Treat raw-text tokenization and request tokenization as different concepts.
Include tool and protocol overhead where possible.
Recalculate when the user switches models.
Apply safety margins based on estimator confidence.
Store provider-reported usage after execution.
Preserve native usage subcategories.
Measure estimator error without logging conversation content.

Where BYOKchat fits

A BYOK client can support many providers without pretending their tokenizers are identical. The context layer can use the best available model-specific estimate, reserve conservative headroom, then let the provider’s returned usage become the source of truth for analytics and cost calculations.

That approach remains reliable even when a user adds a custom OpenAI-compatible model the app has never seen before.