Circuit Breakers for AI Providers

Retries answer:

Should this request try again?

Circuit breakers answer a different question:

Should we keep sending new requests to a target that appears unhealthy?

That distinction matters in AI clients because a provider outage can affect many chats simultaneously. If every request independently retries, the application can create a storm of doomed traffic while making users wait through repeated failures.

A circuit breaker temporarily stops normal dispatch to an unhealthy scope, waits for a cooldown, and then allows limited probes to determine whether service recovered.

The basic state machine

Closed

Requests flow normally while the breaker collects health evidence.

Open

Normal requests are blocked or routed elsewhere according to user policy.

Half-open

A small number of probe requests are allowed.

This prevents the first recovery instant from releasing the entire queued workload at once.

Scope is the hardest design decision

A breaker can be scoped to:

provider
base endpoint
region
account/connection
model
operation class

Choosing too broad a scope creates false outages.

For example:

one model returns "context too long"

should not open the circuit for every model at the provider.

Likewise:

one user's API key is invalid

should not mark the provider globally unhealthy.

Breakers should react to transient service evidence

Candidate trip signals can include:

connection failures
repeated 5xx responses
gateway failures
provider-declared temporary unavailable/capacity errors
repeated first-response timeouts
repeated stream failures attributable to endpoint/service

Signals that usually should not count as provider-health failures include:

401 invalid credential
403 account permission
400 malformed request
context limit exceeded
unsupported capability
user cancellation
tool permission denial
local persistence failure

The error classifier should make this distinction before breaker accounting.

See How to Classify AI API Errors.

A single failure is usually not enough

Networks are noisy.

One transient reset does not prove the provider is down.

Trip policy can use a rolling window such as:

minimum sample count
failure ratio threshold
consecutive severe failures
latency/timeouts above threshold

Avoid copying arbitrary numbers from unrelated systems. The correct thresholds depend on request volume, workload, and how expensive false trips are.

Low-volume clients need different logic

A desktop/mobile BYOK app may send only a handful of requests per minute.

A percentage such as:

50% failure rate

is meaningless if the sample is two requests.

Use a minimum evidence requirement before interpreting ratios.

For low traffic, consecutive failures plus a short cooldown can be simpler and more understandable.

High-volume clients need a rolling window

If a provider handles many simultaneous requests, a breaker should avoid lifetime counters.

Use recent evidence:

last N attempts
or
last T seconds

Old incidents should age out.

A provider that failed yesterday but succeeded all morning should not remain penalized.

Do not count retries as independent user evidence blindly

One logical request can create several retry attempts.

If all three attempts fail during the same outage, they are real endpoint observations—but they can overweight one user’s operation relative to unrelated requests.

Keep both:

attempt-level health signal
logical-request outcome

and define breaker policy deliberately.

429 needs careful scope

Rate limiting can reflect:

account quota
model quota
token rate
request rate
provider capacity

An account-specific 429 should not necessarily trip a provider-wide breaker.

If the provider reports temporary shared capacity pressure, it may be a useful availability signal.

Normalize what the provider actually told you instead of treating all 429s identically.

See Retry-After and Rate-Limit Headers Explained.

Authentication failures should disable a connection, not open a provider circuit

If an API key is invalid, the useful state is:

connection requires credential repair

not:

provider unavailable

The user action is different, the recovery condition is different, and other accounts may work perfectly.

Separate configuration health from service health.

Capability errors should update capability knowledge

If a model rejects an unsupported feature, the client can update or invalidate its capability profile.

That is not a circuit-breaker event.

Examples:

model lacks tool calling
model rejects image input
reasoning effort parameter unsupported
structured output unsupported

See Capability Detection in Multi-Model AI Apps.

Timeouts can be breaker signals only when the phase is understood

A connect timeout or repeated first-response timeout can indicate endpoint trouble.

An overall deadline exceeded after a long successful tool loop says much less about provider health.

A stream idle timeout after the app was suspended says nothing useful about the provider.

Record timeout phase and lifecycle cause before feeding it into health policy.

See Timeout Design for AI Applications.

Opening the circuit should fail fast

When the breaker is open, new operations should not wait through a network timeout to discover the known condition again.

The scheduler can return a local state such as:

provider_temporarily_unavailable
retry_not_before = ...

The UI can then offer:

wait
retry manually
switch model/provider
use local model

according to product policy.

Do not silently fallback across privacy boundaries

A circuit breaker can inform routing, but it should not override user intent.

If a user selected:

local only

and the local server circuit is open, the correct outcome is likely:

Local server unavailable

not sending the prompt to a cloud provider automatically.

Likewise, switching between cloud providers may expose content to a different company.

See AI Provider Fallback and Model Routing.

Cooldown is a recovery gate, not a guarantee

After an open circuit waits for its cooldown, do not assume recovery.

Move to half-open and permit a bounded probe.

The probe should be representative enough to test the failing path but inexpensive and safe.

Choose probes carefully

A generic /models request may prove:

DNS works
TLS works
endpoint responds
credentials may work

but it may not prove generation is healthy.

A synthetic generation probe consumes quota and may not be appropriate for a user-owned BYOK credential.

Often the best half-open probe is simply the next real eligible request, admitted one at a time.

Half-open should limit concurrency

If 20 queued requests all become probes simultaneously, the circuit breaker has not actually protected recovery.

Use a small half-open allowance:

one or few trial requests

Then close only after enough evidence for the application’s needs.

Success can close gradually

For sensitive workloads, one successful request may be enough to close.

For unstable providers, you may require several successful probes or use a health score.

Whatever policy you choose, avoid pretending the confidence is mathematically precise without enough samples.

See Health Scoring AI Providers Without Fake Precision.

Use backoff for repeated open cycles

If a provider repeatedly fails immediately after half-open probes, a fixed 1-second cooldown creates continuous hammering.

Increase the cooldown with a cap:

open 1 -> short cooldown
open 2 -> longer cooldown
open 3 -> longer, bounded cooldown

Jitter can spread probes across clients.

This is similar to retry backoff but applies to the target’s admission state rather than one request.

Circuit state should not live forever

For local applications, persisting circuit state across relaunch can be useful for short incidents, but stale health should expire.

Persisting:

provider unhealthy

forever is dangerous.

If you persist state, store evidence timestamps and a bounded retryNotBefore so relaunch does not freeze a recovered provider indefinitely.

Local servers benefit from circuits too

A local OpenAI-compatible endpoint can fail because:

server stopped
Mac went to sleep
model unloaded
LAN address changed
firewall changed
machine overloaded

A circuit breaker can stop repeated foreground waits while the endpoint is clearly unreachable.

But distinguish:

server unreachable

from:

selected model missing

The latter may be fixed by choosing/loading a model rather than waiting for service recovery.

Breakers and request queues should cooperate

The queue owns dispatch.

The breaker contributes admission state:

Do not make every request independently check an unrelated global boolean after it already consumed a worker slot.

See How to Build an AI Request Queue.

Breaker state should be observable

Useful diagnostic fields include:

scope
state
opened_at
retry_not_before
recent sample count
recent transient failures
last success
last failure category
half-open probes in flight

The UI usually needs only a simplified state.

Developer diagnostics can show the evidence without exposing conversation content.

Avoid a universal “provider down” banner from one client

A local application’s circuit breaker describes this client’s recent observations.

It does not prove the provider has a global outage.

Use language such as:

This connection is temporarily failing

rather than:

Provider X is globally down

unless you have an authoritative external status source.

Health should be scoped to operation class when necessary

A provider can have healthy text generation while a hosted search/tool endpoint is degraded.

If your app combines several services, consider operation-scoped health:

chat generation
file upload
embeddings
hosted tools
background jobs

Do not block healthy paths because an unrelated feature failed.

User-initiated manual retry can be a probe

When the breaker is open and the user explicitly taps Retry, you may allow that request as a half-open probe if policy permits.

Make sure it still respects:

rate-limit not-before time
privacy/routing policy
operation idempotency

Manual intent does not make unsafe replay safe.

Breaker metrics should separate false trips

Track at least:

number of opens
time spent open
half-open success rate
requests failed fast
requests routed elsewhere
manual overrides

Review incidents where the circuit opened due to non-health errors. Those indicate taxonomy/scope problems.

Simulate provider degradation

A deterministic provider simulator can test:

3 consecutive 503s
intermittent reset/success
401 repeated
context overflow repeated
account-specific 429
provider-capacity 429
first-byte timeout
partial stream failure
recovery after cooldown
half-open probe failure
half-open probe success
multiple accounts same provider
one model failing, another healthy

Assert the correct breaker scope and state transition.

A circuit-breaker checklist

• The breaker scope is explicit.
• Only relevant transient/service failures count toward tripping.
• Auth, validation, capability, policy, cancellation, and local app errors do not poison provider health.
• A minimum evidence threshold prevents one noisy failure from tripping broad scope.
• Open state fails fast or routes according to user policy.
• Cooldown leads to bounded half-open probes, not an immediate traffic flood.
• Repeated recovery failures increase cooldown with a cap.
• Rate-limit scope is preserved.
• State expires rather than becoming permanent.
• Queue/routing logic consumes breaker state centrally.
• Metrics describe this client’s observations, not unsupported global outage claims.

Where BYOKchat fits

A BYOK client has many independent connection identities: direct provider accounts, custom OpenAI-compatible endpoints, and local servers. Circuit breakers should therefore live near the connection/request scheduler rather than as one global “provider up/down” flag.

With good error classification, the app can stop hammering an endpoint during genuine transient failure while still letting users repair bad credentials, change incompatible models, or keep using another healthy account on the same provider.

Further reading

• How to Classify AI API Errors
• Health Scoring AI Providers Without Fake Precision
• How to Build an AI Request Queue
• AI Provider Fallback and Model Routing
• Retry-After and Rate-Limit Headers Explained