How to Build an AI Request Queue

An AI client often starts simple:

user taps Send -> start network request

That works until the app can run several kinds of work at once:

multiple chats
retries
background generations
tool loops
file uploads
local models
remote models
MCP calls
analytics/reconciliation

Without a queue, every subsystem can start work independently. The result is easy to recognize:

too many simultaneous requests;
local models fighting for memory;
provider rate limits;
retries competing with fresh user actions;
cancelled chats continuing in the background;
no clear answer to “what is running?”

A request queue turns that uncontrolled concurrency into explicit policy.

The queue should schedule logical work, not raw HTTP callbacks

A useful queue item represents an application operation:

interface AIQueueItem {
  id: string;
  conversationId?: string;
  providerConnectionId: string;
  modelId: string;
  priority: Priority;
  state: QueueState;
  createdAt: Instant;
  deadline?: Instant;
}

It should not merely be:

closure that calls URLSession

Durable identity makes cancellation, recovery, observability, and retry behavior much easier.

Separate logical request state from attempts

A logical queue item can create several provider attempts:

item A
  attempt 1 -> 503
  backoff
  attempt 2 -> success

The queue should keep the user-facing operation stable while attempts come and go underneath it.

This preserves one conversation turn instead of creating duplicate turns for retries.

Define a small state machine

For example:

Background/provider-job workflows may need additional states such as:

waiting_remote
reconciling
input_required

The important part is that transitions are explicit.

Queue concurrency and provider concurrency are different

A global limit such as:

maximum 4 AI jobs

is useful but incomplete.

You may also need scoped limits:

provider A: max 2
provider B: max 4
local model server: max 1
background image jobs: max 2

Different endpoints have different resource constraints.

A local 8B model can saturate a machine with one generation while a cloud provider may comfortably handle several independent requests.

Do not use one concurrency constant as a universal truth.

Connection identity matters

Two accounts for the same provider may have different quota/rate-limit state.

Schedule using a scope such as:

(provider, connection/account, model, operation class)

when the provider semantics justify it.

Otherwise one rate-limited account can unnecessarily block another.

Fresh user work should not starve behind maintenance

Queue priorities can be simple:

interactive foreground
user-requested background
retry/recovery
prefetch/maintenance

A user tapping Send should usually outrank nonessential analytics reconciliation.

But priority does not mean absolute starvation.

A perpetual stream of foreground work should not prevent an old durable job from ever making progress.

Use aging or weighted fairness

One simple approach is to gradually increase effective priority with wait time.

Conceptually:

effectivePriority = basePriority + ageBoost

Another is weighted round-robin across classes.

You do not need a complicated scheduler to avoid starvation, but the policy should be explicit and testable.

Per-conversation ordering can matter

Two sends in the same conversation often should not execute concurrently because turn B depends on turn A’s final state.

A queue can enforce:

one active generation per conversation

while allowing unrelated conversations to run in parallel.

This avoids accidental histories such as:

user A
user B
assistant answer to B
assistant answer to A

unless branching/concurrent turns are a deliberate product feature.

Edits and regenerate actions can invalidate queued work

Suppose a request is queued and the user edits the preceding message.

The old queued request may now be stale.

Queue items should carry enough conversation/version identity to detect:

context changed before execution

Then cancel/rebuild rather than executing against history the user no longer sees.

Cancellation must work in every state

Cancellation is easy only while an item is waiting.

A full design covers:

queued -> remove
waiting retry -> cancel timer
connecting -> cancel network task
streaming -> cancel stream
running tool -> request tool cancellation where safe
background remote job -> send provider cancellation if supported

The queue item should end in a stable cancelled state even if downstream cancellation is best-effort.

Cancellation does not guarantee remote non-execution

If the request already reached the provider, cancellation may only mean:

client stopped waiting

The provider may have already consumed tokens or completed work.

For remote side effects, preserve outcome uncertainty and reconcile when possible.

See Designing Reliable AI Retries.

Retries should re-enter through the scheduler

Do not have a failed request sleep inside its execution slot:

worker holds concurrency slot
sleep 30s
retry

That wastes capacity.

Prefer:

Running -> WaitingRetry
release slot
schedule not-before time
re-enter queue later

The retry still belongs to the same logical request.

Rate-limit delays are queue metadata

A retry can carry:

notBefore = providerRetryNotBefore

The scheduler should not dispatch it before that time.

If many items target the same limited provider, the provider bucket itself may enter a cooldown instead of every item maintaining an unrelated timer.

See Retry-After and Rate-Limit Headers Explained.

Deadlines prevent stale queued work

A request can become useless if it waits too long.

Examples:

interactive chat send queued for 5 minutes
search suggestion requested for an old query
regenerate request after user switched branches

Attach a logical deadline or validity predicate.

Before dispatch:

if expired or superseded -> cancel without network request

Persist only work that should survive process death

Not every queue item needs durable storage.

Useful classes are:

Ephemeral

UI suggestion
short foreground generation
nonessential refresh

Durable

provider background job
important file upload
operation with remote job ID
long-running export

Persisting every tiny task can add unnecessary complexity.

The queue should make durability an explicit property.

Durable items need a serialization-safe operation description

Do not persist arbitrary closures.

Persist data such as:

{
  "operation": "provider_background_response",
  "provider_connection_id": "c1",
  "remote_job_id": "job_123",
  "conversation_id": "chat_42",
  "state": "waiting_remote"
}

On relaunch, a coordinator reconstructs the appropriate handler.

Persist before acknowledging durable creation

For a remote background job, a safe flow is:

create local operation ID
send create request
receive provider job ID
persist provider job ID + state
then update UI as durable

The difficult case is a connection failure where the create outcome is unknown.

That requires idempotency/reconciliation rather than blindly creating another job.

See How Long-Running AI Tasks Work.

Local model queues need resource awareness

For on-device or LAN-hosted inference, concurrency is not just a provider quota issue.

Competing jobs can increase:

memory pressure
thermal load
swap
TTFT
battery use
UI lag

A queue can intentionally serialize expensive local generations even while cloud jobs run concurrently.

Keep local resource policy separate from cloud API rate policy.

Tool calls can use a child scheduler

A model turn may request several independent tools.

The application can run them concurrently only when dependencies and side effects allow it.

A tool scheduler can enforce:

per-tool concurrency
resource locks
approval requirements
external API limits

Then aggregate results back into the parent AI request.

See How to Execute Parallel AI Tool Calls Safely.

Do not hold an AI provider slot while waiting for user approval

If a model emitted a tool call and the user must approve it, the network generation phase may already be complete.

Represent the logical turn as:

waiting_for_approval

and release provider concurrency capacity.

When approval arrives, continue the multi-round workflow.

This avoids tying scheduling resources to human think time.

Priority inversion can happen with locks

Imagine:

low-priority tool holds file lock
high-priority interactive turn needs same lock

The scheduler should at least make such waits observable.

For more complex resource managers, priority inheritance can be considered, but many AI apps can avoid the problem by keeping critical sections small and locks narrowly scoped.

Backpressure belongs at admission

If the queue is already saturated, you need a policy for new work.

Possibilities include:

accept and show queued state
reject nonessential background work
coalesce duplicate refreshes
replace obsolete requests
limit total pending items

Unlimited in-memory queues turn temporary provider slowness into an eventual memory/latency problem.

Coalesce duplicate work where semantics allow

Examples:

refresh model list for same connection
health probe for same endpoint
rebuild same derived index

Ten callers can share one in-flight operation instead of creating ten identical requests.

Do not coalesce user chat generations merely because their text happens to match; they have distinct conversation semantics.

Queue state should be visible to the UI

Useful user-visible states include:

Queued
Waiting for provider
Generating
Waiting for tool approval
Using tool
Retrying in 4s
Waiting for background result

Avoid a spinner that hides whether the request has even started.

Developer diagnostics can show deeper scheduler metadata.

Queue wait belongs in latency metrics

If the user waits 3 seconds locally before a provider request starts, measuring only provider duration understates perceived latency.

Track:

queue_wait_ms
attempt_start
TTFT from attempt start
TTFT from user action
overall_duration

Both technical and user-perceived timing are useful.

Circuit breakers should feed admission decisions

If a provider circuit is open, the scheduler can avoid dispatching doomed requests until the probe window.

But circuit scope matters.

A model-specific failure should not necessarily block all models on the provider.

See Circuit Breakers for AI Providers.

Fairness across providers improves responsiveness

Suppose provider A has 100 queued background jobs and provider B has one interactive request.

A single FIFO queue with a small global concurrency cap can let A dominate.

Use independent provider buckets or a scheduler that considers provider scope so unrelated endpoints do not block each other unnecessarily.

Avoid queue deadlocks in multi-round workflows

A classic mistake:

parent AI request holds only provider slot
parent waits for child AI request on same provider
child cannot start because slot is held

Release or re-entrant-design scheduling resources at phase boundaries.

The logical operation can remain active without monopolizing every underlying execution slot.

Recovery after app relaunch should reconcile, not restart blindly

For durable items:

load queue journal
inspect last known state
query remote status where possible
resume local processing
mark impossible operations for user review

Do not treat every running record from the previous process as proof that the remote operation failed.

A crash can occur after remote completion but before local persistence.

Keep a bounded operation journal

Useful records include:

logical request ID
state transitions
attempt IDs
provider request IDs
remote job IDs
retry schedule
cancellation reason
final outcome

You do not need to persist prompts/responses in the scheduler log if the conversation store already owns them.

Keep telemetry content-minimized.

A scheduler architecture

Durable operations additionally write state to persistent storage around important transitions.

Testing the queue

Use deterministic fake providers and clocks.

Cover:

global concurrency cap
per-provider cap
per-conversation serialization
priority ordering
aging/fairness
cancel while queued
cancel while streaming
retry releases slot
retry not-before respected
rate-limited provider bucket
expired request never executes
provider A does not starve provider B
local-model serialization
app crash + durable recovery
remote job reconciliation
queue overflow/coalescing

Assert that no operation starts more times than intended.

A queue checklist

Every logical request has stable identity.
Provider attempts are separate from logical operations.
Concurrency is scoped appropriately.
Same-conversation ordering is explicit.
Queue priorities cannot starve lower classes forever.
Cancellation works while queued, retrying, and running.
Retries release execution slots while waiting.
Rate-limit not-before times feed the scheduler.
Pending work is bounded or coalesced.
Durable work is serializable and recoverable.
Queue wait contributes to user-perceived latency metrics.
Remote unknown outcomes are reconciled rather than blindly replayed.

Where BYOKchat fits

A multi-provider AI client naturally becomes a scheduler once it can run several chats, local models, remote providers, tool loops, and background operations at once.

A shared queue can enforce per-conversation ordering, provider/account concurrency, local-model resource limits, retry timing, cancellation, and durability while leaving provider-specific request construction inside adapters.

That architecture keeps performance predictable without forcing every feature to invent its own concurrency rules.