How to Design Context Management for Long AI Conversations

A long AI conversation eventually contains more information than should—or can—be sent to the model on every request. Reliable chat software therefore needs a context manager, not just a growing messages array.

The context manager answers a concrete question for each generation:

Given this conversation, this model, this tool set, and this token budget, what information should be sent now?

That is different from deciding what the application should persist. Durable history can be large. Active model context must be deliberately constructed.

Start with a budget, not truncation

A model context window is shared by multiple categories of input and output.

A simplified budget is:

$$B_{input} = C - R_{output} - R_{overhead}$$

where:

• $C$ is the model’s usable context capacity;
• $R_{output}$ reserves room for generation;
• $R_{overhead}$ reserves protocol/tool/system overhead;
• $B_{input}$ is what remains for assembled input.

The exact provider accounting can differ, but the architectural lesson is stable: do not fill the context to its maximum with history and then hope there is room for the answer.

Durable history and active context are different products

Persist everything the product needs:

conversation database
├── user messages
├── assistant messages
├── tool calls/results
├── attachments
├── edits/branches
└── generation metadata

Then derive a smaller request:

active context
├── instructions
├── relevant project context
├── selected tool definitions
├── recent conversation
├── older compressed context
└── current user request

Never delete durable user history merely because it no longer fits the current model request.

Priority beats “last N messages”

Keeping only the newest messages is easy but often wrong.

Imagine the user said near the beginning:

This project must support iOS 26 and macOS 26, and never use a server-side database.

Forty turns later, those constraints may be more important than several recent debugging exchanges.

A useful context system can classify information by role and importance:

hard instructions
current task
explicit user constraints
recent conversational dependencies
relevant tool evidence
older background
low-value chatter

Recency is one signal, not the entire policy.

Keep system and user authority distinct

Compaction should not flatten instruction hierarchy into an ambiguous paragraph.

If system/developer/project instructions and user conversation are merged into one generated summary, the model may lose important authority boundaries.

Preserve instruction layers explicitly where the provider protocol supports them. Summarize conversational content separately from higher-priority rules.

Reserve output before assembling input

If a user asks for a long code review, the response may need substantially more output room than a yes/no answer.

A context manager can choose an output reserve based on task type or configured maximum.

Conceptually:

const inputBudget = contextLimit
  - desiredOutputReserve
  - toolAndProtocolReserve;

If the provider exposes a maximum-output parameter, that can make the boundary explicit. But the application still needs to budget conservatively for provider-specific accounting.

Tool definitions consume context too

A chat with 50 tools can spend a meaningful amount of context describing capabilities before any conversation history is included.

This is one reason per-chat tool selection helps both safety and efficiency.

Instead of exposing every connected capability:

all MCP servers × all tools × every chat

prefer the tools relevant to the current workflow.

Smaller stable tool sets can also improve prompt-cache reuse.

Tool results can dominate a conversation

A search tool might return 30 documents. A database tool might return thousands of rows. A browser tool might return an entire page.

Blindly appending raw results causes context growth much faster than ordinary chat text.

Control tool evidence with:

• result-size limits;
• pagination;
• field selection;
• ranking;
• references to durable results;
• extraction of relevant passages;
• safe summarization when appropriate.

The model needs enough evidence to answer, not necessarily the entire underlying dataset.

Attachments need a lifecycle

An attachment can affect context in several ways:

1. provider file reference;
2. extracted text;
3. image input;
4. local searchable chunks;
5. summary or metadata.

Do not assume the full attachment must be resent forever after it first appears.

A durable chat can remember the attachment while the context manager chooses the representation needed for the current turn.

Sliding windows are the simplest baseline

A basic strategy is:

1. include required instructions;
2. include the current user turn;
3. walk backward through prior turns until the budget is full.

This is easy to reason about and preserves recent local coherence.

Its weakness is forgetting important older facts.

For short chats, that may be acceptable. For long-running projects, you usually need another layer.

Summaries create compressed memory

A conversation summary can preserve older information in fewer tokens.

For example:

Turns 1–80
  ↓
structured summary
  +
turns 81–100 verbatim
  +
new user message

But summaries are lossy. They can omit details, introduce errors, or blur who said what.

Treat them as derived artifacts, not replacements for durable history.

Prefer structured summaries for important workflows

Instead of one prose blob, a summary can preserve categories:

user_goals:
  - ship universal iOS/macOS app
constraints:
  - local-first data
  - no server account required
decisions:
  - provider credentials stay in Keychain
open_questions:
  - backup merge semantics
completed_work:
  - provider abstraction implemented

Structured summaries make it easier to update one category without rewriting everything and reduce accidental loss of durable constraints.

Summaries need provenance

When a model acts on compressed history, debugging is easier if the application knows what the summary covers.

Store metadata such as:

summary version
covered turn range
creation model
created timestamp
source conversation revision

If the user edits a message inside the covered range, the summary may be stale and should be invalidated or regenerated.

Do not recursively summarize forever

A common degradation pattern is:

history → summary A
summary A + more history → summary B
summary B + more history → summary C

Each generation can compound omissions.

Where practical, regenerate from canonical source ranges or maintain structured state whose fields can be reconciled against source turns.

Long-lived summaries deserve the same skepticism as any generated content.

Compaction can be more than summarization

Some context can be reduced deterministically:

• remove redundant protocol metadata;
• omit completed low-value tool payloads;
• replace large result bodies with stable references plus selected excerpts;
• remove superseded drafts;
• deduplicate repeated attachment descriptions;
• include only enabled tool schemas.

Use deterministic compaction before asking another model to rewrite information whenever possible.

Model switching changes the budget

A conversation prepared for one model may not fit another.

If Model A has a larger usable context than Model B, switching to B can require immediate compaction.

The context manager should therefore take the selected model as an input:

buildContext(conversation, modelCapabilities, request)

not build one universal request and send it everywhere.

See Reliable AI Provider Fallback and Model Routing.

Tokenizers are not universally interchangeable

Different model families can tokenize the same text differently. Even providers exposing similar OpenAI-compatible schemas may not share exact token accounting.

Do not rely on one local tokenizer as perfect truth for every provider.

Useful approaches include:

• provider usage from completed requests;
• model-specific tokenizers where available;
• conservative estimates;
• safety margins;
• retry/rebuild on explicit context-length errors.

The goal is reliable budgeting, not pretending an estimate is exact.

Prompt caching affects context layout

Caching often rewards stable prefixes. Context management that constantly rewrites the beginning of a prompt can reduce cache reuse.

A cache-friendly layout tends to keep stable material early:

stable instructions
stable tool definitions
stable project context
older stable conversation/summary
recent changing turns
new user input

Provider caching semantics differ, so do not distort correctness merely to chase cache hits. But context stability can be a useful optimization after semantics are correct.

See AI Prompt Caching Explained.

Reasoning state may consume context without appearing as chat text

Reasoning-capable APIs can have provider-specific continuation state that is not equivalent to visible assistant text.

Your context manager should not assume:

visible characters = all context usage

Provider-reported usage is the best authority when available. Opaque reasoning items may also need to be preserved through the adapter for correct continuation even though the product does not render them as normal messages.

Branches and edits invalidate derived context

If the user edits turn 20 and regenerates, summaries derived from turns 20–80 may no longer describe the active branch.

Derived artifacts should be associated with a conversation revision or branch.

This applies to:

• summaries;
• embeddings/indexes;
• provider continuation IDs;
• cached context calculations;
• selected evidence.

A chat tree is not always a single immutable list.

Context errors should trigger rebuilding, not blind retry

If the provider rejects a request because it exceeds context capacity, sending the identical request again is pointless.

Classify the failure and rebuild with a tighter budget:

Bound this loop. A bug in estimation should not create infinite retries.

Preserve the current task aggressively

When trimming, never lose the message the user is asking you to answer.

That sounds obvious, but naive “drop oldest tokens until under limit” algorithms can damage multi-part current inputs, attachment references, or tool-result dependencies.

Build context in semantic units—messages, blocks, tool rounds—not arbitrary string slices.

Keep tool-call/result pairs coherent

Provider protocols often require tool calls and their corresponding results to remain structurally consistent.

Do not trim away the call while leaving the result, or vice versa, if the target API expects both.

Treat a completed tool round as an atomic context unit unless the adapter has a defined way to translate it into neutral evidence.

A practical assembly algorithm

One architecture is:

The exact ranking policy is product-specific, but making the stages explicit is more maintainable than scattered if tokenCount > ... checks.

Measure context behavior

Useful telemetry can remain content-free while tracking:

estimated input tokens
provider-reported input tokens
output reserve
number of included turns
number of omitted turns
summary tokens
number of tool schemas
attachment/evidence tokens
compaction triggered
context-length failures
cache-read tokens where reported

This reveals whether your context policy works without logging the user’s prompts.

Test pathological conversations

A context manager should be tested with:

• one huge user message;
• many tiny messages;
• giant code blocks;
• large tool results;
• dozens of tool definitions;
• attachments;
• reasoning models;
• model switch to a smaller context;
• edited old turns;
• stale summaries;
• multilingual text;
• context-length provider errors;
• a current request that depends on a very old constraint.

The hardest bugs appear when several categories compete for the final few thousand tokens.

A context-management checklist

Before calling long chats reliable, verify that:

• persistence and active context are separate;
• output space is reserved before input assembly;
• instructions have explicit priority;
• recent history is not the only relevance signal;
• tool definitions and results are budgeted;
• attachments have a context lifecycle;
• summaries are treated as lossy derived data;
• edits invalidate stale derived context;
• model switching recalculates the budget;
• token estimates use safety margins;
• tool call/result structure remains valid;
• context-length errors trigger bounded rebuilding;
• telemetry measures context behavior without storing content.

Where BYOKchat fits

A multi-provider client should keep one durable conversation while building a provider/model-specific context for each generation. That lets the user change models without changing the stored chat format, and lets context policy evolve independently from persistence.

The context window is a runtime constraint. It should not define what the user is allowed to keep in their conversation history.

Further reading

• What Is an AI Context Window?
• Stateful vs Stateless AI Conversations
• AI Prompt Caching Explained
• How a BYOK AI Chat Request Works