AI Tokens Explained for Developers

AI APIs do not usually bill or limit requests by characters or words. They operate on tokens: model-specific units produced by a tokenizer before text reaches the model.

A token might be:

• an entire short word;
• part of a longer word;
• punctuation;
• whitespace combined with text;
• a fragment of code;
• part of a non-Latin script;
• a special control symbol used by the model protocol.

That means this intuition is wrong:

1 word = 1 token

A better mental model is:

text
  ↓
tokenizer
  ↓
model-specific token IDs
  ↓
model

Token counts matter because they affect context limits, output limits, cost estimation, truncation, prompt caching, and how much conversation history your application can send.

A token is not a word

Consider these strings:

cat
internationalization
hello_world
🙂

A tokenizer may represent each with a different number of tokens. Even similar-looking strings can tokenize differently depending on spacing, capitalization, punctuation, or language.

For example, these can produce different token boundaries:

Hello
 hello
HELLO
hello!

The tokenizer is optimized for the model’s training and vocabulary, not for making human-readable boundaries.

Why tokenizers use subwords

A fixed vocabulary containing every possible word would be impractical.

Instead, modern language models typically use vocabularies that can represent common sequences directly while composing rarer words from smaller pieces.

Conceptually:

"unbelievable"
→ "un" + "believ" + "able"

The actual pieces depend on the tokenizer.

Subword tokenization gives the model a manageable vocabulary while still allowing it to represent:

• rare words;
• new names;
• code identifiers;
• misspellings;
• multiple languages;
• arbitrary text.

Tokens become integer IDs

The model does not receive the original text exactly as a sequence of characters.

The tokenizer maps text into integer IDs:

"hello world"
→ [1234, 5678]

Those numbers index the model’s vocabulary.

The exact IDs are model-specific and are not semantically meaningful to your application. What matters to most client software is the count and how the provider accounts for it.

Input tokens and output tokens are different

A request usually has at least two usage categories:

input tokens
output tokens

Input tokens can include more than the user’s visible message.

For a chat request, input may contain:

system/developer instructions
conversation history
current user message
tool definitions
retrieved documents
attachment text
provider protocol overhead
reasoning continuation state

Output tokens are what the model generates for that request, which may include visible answer text and provider-specific reasoning or tool-call content depending on the API.

Visible text is not necessarily total token usage

A user may see a 50-token answer while the provider reports substantially more generated usage.

Reasoning-capable models can consume tokens for internal reasoning that are not rendered as normal answer text. Tool-capable APIs can also generate structured call arguments or other items.

So this is unsafe:

usage = tokenize(visibleAssistantText)

Provider-reported usage is the authoritative source after a request completes.

Context windows are measured in tokens

A model context limit defines how much model-visible information can participate in a request or conversation state.

A simplified budget is:

$$C \ge I + H + T + R + O$$

where:

• $C$ = usable context capacity;
• $I$ = instructions/current input;
• $H$ = conversation history;
• $T$ = tool schemas and tool results;
• $R$ = retrieval/attachments/other context;
• $O$ = output or reasoning reserve, depending on provider semantics.

The exact accounting rules vary by model and API.

The practical lesson is stable: do not fill the entire advertised context with input and assume the model can still generate an arbitrarily long answer.

See What Is an AI Context Window?.

Output limits are also token-based

Most AI APIs expose some kind of maximum generated-token setting.

Depending on the provider, the control may bound:

• visible output tokens;
• total generated tokens including reasoning;
• a provider-specific notion of completion tokens.

Do not normalize every provider field into one misleading label without understanding the native semantics.

We cover this distinction in Context Window vs Output Limit vs Reasoning Tokens.

Token counts depend on the tokenizer

Two models can count the same text differently.

Suppose your app estimates:

prompt = 8,000 tokens

using Model A’s tokenizer.

Model B may see the same text as:

7,300 tokens

or:

9,200 tokens

That makes a universal exact token counter impossible across arbitrary providers and model families.

See Why Different AI Models Count Tokens Differently.

English rules of thumb are only rough estimates

Developers often use shortcuts such as:

1 token ≈ 4 English characters
1 token ≈ 0.75 English words

These can be useful for rough planning, but they are not guarantees.

They can fail badly for:

• source code;
• JSON;
• random IDs;
• URLs;
• tables;
• CJK text;
• mixed-language text;
• emoji;
• unusual Unicode;
• compressed/minified text.

Never use an English prose heuristic as an exact context-limit guard.

Code can tokenize very differently from prose

Consider:

func loadConversation(id: UUID) async throws -> Conversation

The tokenizer may split punctuation, identifiers, keywords, spaces, and common substrings differently from English sentences.

Long generated code can therefore have a token/character ratio that differs materially from normal prose.

This matters when building:

• code assistants;
• diff reviewers;
• repository chat;
• JSON-generating agents;
• structured-output workflows.

Measure representative inputs rather than relying on prose assumptions.

URLs and IDs can be unexpectedly expensive

A long natural-language phrase may contain many reusable vocabulary pieces.

A random-looking string such as:

6e0f74cf-bbb0-44f1-8d3d-1b3594ca21ad

may break into many smaller tokens because it does not match common vocabulary patterns.

The same applies to:

• hashes;
• opaque database IDs;
• signed URLs;
• base64;
• compressed text;
• minified bundles.

Do not put large opaque blobs into model context unless the model actually needs them.

Images and audio may use non-text token accounting

Multimodal APIs can convert images, audio, or video into model-specific units that are billed or constrained alongside text.

The provider may report these as:

• image tokens;
• audio tokens;
• input tokens with modality details;
• a separate usage category.

Do not assume tokenize(extractedText) approximates the cost of an image request.

Use provider documentation and returned usage metadata for the selected model.

Tool schemas consume tokens

Tool-capable requests often include JSON Schema-like definitions:

{
  "name": "search_docs",
  "description": "Search project documentation",
  "parameters": {
    "type": "object",
    "properties": {
      "query": { "type": "string" }
    },
    "required": ["query"]
  }
}

That schema becomes part of the model’s request context.

If your client exposes 100 verbose tools on every turn, tool metadata can consume substantial input tokens before the user message is considered.

This is one reason capability/tool selection should be deliberate.

Retrieved documents consume tokens after retrieval

RAG does not bypass context limits.

Retrieval chooses evidence, then the chosen chunks still need to reach the model in some form.

Conceptually:

100,000 documents
      ↓ retrieve
5 relevant chunks
      ↓
model context

The index can be enormous, but only the selected evidence contributes to the active model context.

See RAG Explained.

Cached tokens are still tokens

Prompt caching can change billing or latency without changing the logical size of the request.

A cached prefix may still be reported as input tokens, with a subcategory describing how many were cache hits.

So these are different questions:

How many tokens are in the request?
How many input tokens were served from cache?
How much did the provider charge?

Do not collapse them into one number.

See AI Prompt Caching Explained.

Token usage is usually known exactly only after execution

Before sending, the client can estimate.

After completion, the provider can report actual usage according to its native accounting.

A robust architecture distinguishes:

estimated usage
provider-reported usage

For example:

type UsageRecord = {
  estimatedInputTokens?: number;
  reportedInputTokens?: number;
  reportedOutputTokens?: number;
  reportedReasoningTokens?: number;
  cachedInputTokens?: number;
};

Not every provider exposes every field. Keep missing data explicit rather than inventing zeroes.

Why exact preflight counting is difficult in multi-provider apps

A client may not have:

• the exact tokenizer implementation;
• the exact model revision;
• provider-injected protocol tokens;
• hidden tool formatting;
• image/audio tokenization rules;
• server-side continuation state size;
• reasoning accounting details.

Therefore a portable client often needs a conservative strategy:

best available model-specific counter
        ↓
provider-specific overhead estimate
        ↓
safety margin
        ↓
request
        ↓
replace estimate with reported usage

Safety margins are better than false precision

Suppose you estimate an input at 126,800 tokens for a model with a 128,000-token limit.

Even if the estimator is usually accurate, sending the request with almost no margin can fail because of protocol overhead or tokenizer mismatch.

A better context manager reserves space for:

• output;
• tools;
• provider wrappers;
• estimation uncertainty.

The right margin depends on the provider and request type.

Token counting should happen after context assembly

Do not count the database conversation and assume that is the request size.

The actual flow should be closer to:

The same persisted chat can produce different token counts for different models.

Token counts should not define persistence

A common design mistake is deleting old messages because they no longer fit the selected model.

Persistence and active context are separate concerns.

Keep the user’s durable conversation, then build a model-specific subset for each generation.

See How to Design Context Management for Long AI Conversations.

Cost is not simply token count

Token count is an input to cost calculation, but pricing can vary by:

• model;
• input vs output;
• cached vs uncached input;
• batch/background mode;
• reasoning accounting;
• provider route;
• modality.

A general cost model is:

$$Cost = \sum_i Tokens_i \times Rate_i$$

where each $i$ represents a separately priced usage category.

Do not hard-code one tokens × price formula for every provider.

See Understanding AI API Costs.

Token metrics are useful without logging content

You can measure:

input tokens
output tokens
reasoning tokens when reported
cached tokens
context utilization percentage
estimated-vs-reported error

without storing the user’s actual prompts or answers.

This is valuable for local/private analytics because it helps diagnose cost and performance while preserving content privacy.

Common token-counting mistakes

Treating words as tokens

Useful for a quick mental estimate; unsafe for enforcement.

Using one tokenizer for every model

Good enough only if you explicitly accept estimation error.

Ignoring tool schemas

Can cause requests to exceed the context limit unexpectedly.

Counting only visible messages

Misses system instructions, retrieval, attachments, provider state, and tools.

Assuming streamed text length equals provider usage

Reasoning and structured output can make reported usage larger.

Trusting a preflight count more than provider usage

After execution, provider-reported usage is the better accounting source.

A practical implementation checklist

For production token accounting:

• keep model-specific tokenization behind a capability/service boundary;
• distinguish estimates from reported usage;
• include tools, retrieval, instructions, and attachments in preflight accounting;
• reserve output space before filling input context;
• use safety margins;
• preserve provider usage subcategories where available;
• do not treat cached input as zero tokens;
• do not delete durable conversation history to satisfy a runtime context limit;
• measure estimation error over time;
• test code, multilingual text, URLs, IDs, JSON, and other non-prose inputs.

Where BYOKchat fits

A multi-provider client needs token accounting at the request layer rather than the conversation-storage layer. Each generation can estimate against the selected provider/model, then persist provider-reported usage after completion for cost and performance analytics.

That keeps token budgets accurate enough to operate without making one model’s tokenizer define the application’s entire data model.

Further reading

• What Is an AI Context Window?
• Why Different AI Models Count Tokens Differently
• Understanding AI API Costs
• How to Design Context Management for Long AI Conversations
• AI Prompt Caching Explained