Health Scoring AI Providers Without Fake Precision

A dashboard that says:

Provider A health: 93
Provider B health: 81

looks precise.

It may communicate almost nothing.

What does 93 mean if:

only three requests were observed;
one account has an invalid API key;
the provider’s text model is healthy but file upload is failing;
latency increased because prompts became much larger;
all measurements came from one network;
the score mixes user cancellations with server failures?

A useful health model should make uncertainty and scope explicit instead of compressing everything into a decorative number.

Start by defining what “health” means

For an AI client, health can include several dimensions:

reachability
request success
stream stability
latency
rate-limit pressure
model/capability availability
background-job completion
tool/service availability

These dimensions do not always move together.

A provider can be reachable but slow. A model can return successfully but stream unreliably. One account can be rate-limited while another works.

Health is always scoped

Record health for the smallest useful unit:

provider
connection/account
base endpoint
model
operation class
network path

For a BYOK client, connection/account is especially important because users can configure multiple accounts or custom endpoints that share the same provider family.

A bad credential on one connection says nothing about another.

Separate configuration validity from service health

These events should usually not reduce service health:

invalid API key
missing credential
unsupported model parameter
context too long
invalid tool schema
user cancellation
local database failure

They are actionable failures, but not evidence that the remote provider is unhealthy.

The error taxonomy must feed health scoring correctly.

See How to Classify AI API Errors.

Use recent windows, not lifetime averages

A lifetime success rate hides current incidents.

Suppose:

99,900 historical successes
100 failures in the last minute

The lifetime success rate still looks excellent while the provider may currently be unusable.

Use a rolling window:

last N eligible attempts
or
last T minutes

and let old evidence age out.

Keep sample size next to every rate

This:

success rate: 50%

means something very different for:

1 success / 2 attempts

than for:

5,000 successes / 10,000 attempts

Never present a rate without the sample context in developer diagnostics.

For user-facing routing, require minimum evidence before letting a noisy rate dominate decisions.

Low-volume clients should avoid pseudo-statistics

An indie desktop/mobile AI app may only issue a few requests per hour.

A fancy exponentially weighted score with four decimal places does not create information that is not there.

A simpler state can be more truthful:

Healthy: recent requests succeeded
Degraded: repeated transient failures
Unavailable: circuit open
Unknown: insufficient recent evidence

You can still track exact metrics underneath it.

Success rate needs an eligible denominator

Decide which outcomes count toward provider health.

Possible eligible failures:

connection refused
repeated TLS/network failure attributable to endpoint
5xx
provider temporary unavailable
first-response timeout
unexpected stream termination

Usually exclude:

user cancellation
invalid request
context overflow
auth/permission issue
unsupported capability
local persistence failure

Otherwise users who cancel many generations can make a healthy provider look bad.

Track transport and semantic completion separately

A streaming request can:

connect successfully
receive 300 tokens
then terminate unexpectedly

If you count only HTTP status, this may appear successful.

Useful dimensions include:

connection success
first response success
semantic completion success
stream interruption rate

This is much more informative than one success boolean.

Latency is a distribution

Avoid using only average latency.

One slow outlier can distort it, and averages hide tail behavior.

Useful measures include:

median
p90
p95
recent max
TTFT distribution
overall duration distribution

For low sample counts, simply show recent values instead of pretending percentiles are meaningful.

Compare like with like

Latency depends heavily on:

model
prompt size
output size
reasoning effort
tool usage
cache state
network path

Do not conclude:

provider A is slower than provider B

from requests with completely different workloads.

For routing, latency can be treated as a recent operational hint after capability and user policy are satisfied—not as a universal quality ranking.

TTFT and generation speed are different dimensions

A provider can have:

slow TTFT + fast output

or:

fast TTFT + slow output

Track both if performance matters.

See Measuring Time to First Token Correctly and AI Generation Speed Explained.

Rate-limit pressure is not exactly availability

An account can be healthy but temporarily quota-limited.

Useful rate-limit signals include:

recent 429 frequency
provider retry-not-before
remaining request/token metadata when documented
queue delay caused by provider limits

Keep the scope at the correct account/model/quota bucket.

Do not turn one user’s quota exhaustion into a global provider incident.

Circuit state is a strong signal, not the whole score

A circuit breaker aggregates recent transient failures and can expose:

closed
open
half-open

This state should strongly influence admission/routing while active.

But circuit state does not replace richer observability, and it should expire/recover with new evidence.

See Circuit Breakers for AI Providers.

Prefer an explainable health snapshot

Instead of:

{ "health": 87.4 }

consider:

{
  "state": "degraded",
  "scope": "connection:anthropic-personal",
  "samples": 18,
  "transient_failures": 5,
  "completed_streams": 12,
  "interrupted_streams": 3,
  "median_ttft_ms": 950,
  "circuit": "closed",
  "observed_since": "..."
}

Now a developer can understand why routing considered the target degraded.

If you need one scalar, expose its ingredients

Sometimes ranking code wants a comparable value.

You can compute one internally, but keep it humble.

For example:

score = reliability component
      + latency component
      - active rate-limit penalty
      - circuit penalty

Then attach:

sample count
window
input signals
confidence band/state

Do not present 84.7 to users as if it were a scientifically measured provider quality score.

Capability is a filter, not a health bonus

A model either satisfies required features or it does not.

If the request requires:

image input
tool calling
strict structured output
specific context size

filter incompatible targets first.

Do not let a high health score cause routing to a model that cannot execute the request.

See Capability Detection in Multi-Model AI Apps.

Privacy policy is also a hard constraint

If a chat is local-only, a healthy cloud provider is still not an eligible route.

Routing order should look more like:

1. hard capability constraints
2. privacy/user policy
3. current availability/circuit state
4. preference/latency/cost signals

Health is an input, not permission.

Separate provider quality from connection health

A BYOK app can know:

this endpoint has recently succeeded from this device

It cannot infer broad statements such as:

this provider is globally 99.99% available

without external, representative data.

Label local observations as local observations.

Network changes invalidate some evidence

A mobile device can switch:

Wi-Fi -> cellular
VPN on -> VPN off
home LAN -> public network

A local server unreachable on cellular does not mean it became unhealthy.

Health state for private/local endpoints should consider network context and expire aggressively across major network-path changes.

Local model health needs resource signals

For local servers, useful evidence can include:

endpoint reachable
model loaded/model available
queue depth
memory pressure
recent TTFT
recent generation speed
thermal or device constraints when available

Do not convert these into fake model “intelligence” scores.

They are operational signals only.

Confidence can be qualitative

A simple confidence field can be:

unknown
low
medium
high

based on sample count and recency.

For example:

2 recent requests -> low confidence
50 consistent recent requests -> higher confidence
no requests after network change -> unknown

The exact thresholds are product decisions and should be tested, not treated as universal constants.

Avoid feedback loops in routing

If your router always sends traffic to the currently “healthiest” provider, other providers stop receiving samples.

Their scores become stale, which makes the router even more likely to keep choosing the same provider.

Possible mitigations include:

explicit user choice remains primary
health only influences fallback
stale health decays toward unknown
half-open probes refresh circuits
occasional safe probe traffic where appropriate

Do not create hidden exploration traffic that consumes user BYOK quota without a clear product reason.

Measure user-perceived and attempt-level latency separately

If a request spends 4 seconds queued locally, then gets a first token in 500ms, both facts matter:

attempt TTFT = 500ms
user-action-to-first-token = 4500ms

Provider health should generally use attempt-level timing.

Product responsiveness should include queue wait.

See How to Build an AI Request Queue.

Tool failures should not poison model health blindly

If the model successfully requests a tool but the user’s calendar API fails, the AI provider may be healthy.

Track:

model round outcome
tool execution outcome
follow-up round outcome

instead of one flattened “agent failed” metric.

This lets you diagnose the correct dependency.

Observability needs stable dimensions

Useful health events can include:

logical_request_id
attempt_id
provider_connection_id
model_id
operation_class
error_category
http_status
TTFT
duration
stream_finish_state
retry_attempt
circuit_state

Avoid high-cardinality sensitive fields such as prompt text or full private URLs.

See Observability for Streaming AI Requests.

A practical health state derivation

One simple approach:

if circuit open:
    Unavailable
else if no recent eligible samples:
    Unknown
else if repeated transient failures or severe recent tail latency:
    Degraded
else if recent eligible requests mostly complete normally:
    Healthy
else:
    Unknown/Degraded depending on evidence

This is less visually impressive than a three-digit score and usually more honest.

Do not let health silently override model choice

If the user explicitly selected a provider/model, health can inform UX:

This connection is currently degraded. Retry anyway or switch?

For automatic fallback, follow the user’s configured policy.

A “smart router” that silently sends prompts elsewhere because one score changed is difficult to trust.

Test with controlled traces

Feed deterministic event sequences:

20 successes
2 network resets
5 context overflows
3 user cancellations
5 account-specific 429s
10 slow but successful TTFTs
3 partial stream failures
network path changes
circuit opens and recovers

Assert which events enter health calculations and what scope changes.

Also test tiny sample counts so a single observation does not produce absurd confidence.

A health-model checklist

Health has explicit scope.
Configuration, auth, capability, policy, cancellation, and local app errors are separated from service health.
Recent windows replace lifetime averages.
Rates carry sample counts.
Latency uses distributions/recent values rather than only averages.
TTFT, stream completion, and generation speed are distinct dimensions.
Account-specific rate limits do not become global incidents.
Circuit state participates in health but is not permanent.
Stale evidence decays toward unknown.
Routing filters capabilities/privacy before considering health.
User-facing labels avoid unsupported global outage claims.
Any scalar score remains explainable and confidence-aware.

Where BYOKchat fits

A multi-provider BYOK client can build useful local health snapshots without collecting prompt content or pretending to operate a global status service. Each connection/model can accumulate recent sanitized request outcomes, TTFT, stream completion, rate-limit state, and circuit state.

The app can then surface “healthy / degraded / unavailable / unknown” diagnostics and use those signals for explicit fallback policies while leaving final provider control with the user.