How AI Model Discovery APIs Work

A model picker looks simple until you support more than one AI provider.

The obvious implementation is:

GET /models
→ display every returned ID

That works for a demo. Production clients need more.

A model-discovery system must answer several different questions:

Which models exist?
Which models can this credential access?
Which models should the app show?
Which capabilities does each model support?
How fresh is this information?
What happens when discovery fails?

Those are separate concerns.

Discovery is not capability detection

A provider may return a model object such as:

{
  "id": "example-model",
  "object": "model",
  "created": 1234567890,
  "owned_by": "provider"
}

That tells the client the model exists.

It may not tell you whether the model supports:

tools;
reasoning;
image input;
structured output;
prompt caching;
a particular context limit;
a particular output limit;
streaming;
hosted tools.

Treat model inventory and model capability metadata as different layers.

Common discovery shapes

OpenAI-style services commonly expose:

GET /v1/models

The response often contains a data array of model objects.

Other providers expose richer catalog endpoints or entirely different APIs.

OpenRouter’s current models API, for example, exposes model properties and supports filtering by supported parameters and output modalities. NVIDIA NIM exposes /v1/models for models currently available to the deployment. These are both useful, but they answer different operational questions.

A multi-provider client should normalize the result into its own model record rather than make the UI consume raw provider objects.

Build an application model record

A useful internal type might look like:

interface DiscoveredModel {
  providerID: string;
  modelID: string;
  displayName: string;
  source: "remote" | "builtIn" | "manual";
  discoveredAt: Date;
  capabilities: ModelCapabilities;
  rawMetadata?: unknown;
}

The important separation is:

provider model ID
≠ display label
≠ capability profile
≠ user preference

Credentials can change the visible catalog

Model discovery is often credential-scoped.

Two API keys for the same provider may see different models because of:

account entitlements;
region;
project/workspace membership;
private fine-tunes;
enterprise access;
staged rollouts;
provider policy.

Therefore cache keys should usually include the logical connection/account, not only the provider name.

Bad cache key:

models:openai

Better:

models:<connection-id>

Do not put the API key itself in the cache key or logs.

Local servers make discovery more dynamic

A local OpenAI-compatible server may expose only currently loaded or installed models.

That means the list can change when the user:

downloads a model;
unloads a model;
renames an alias;
starts a different server;
changes the base URL.

The UI should make refresh easy and should not treat an old local catalog as permanent truth.

Discovery should degrade gracefully

A failed /models request should not necessarily make the connection unusable.

Some custom endpoints support generation but not model listing. Others require users to type a model ID manually.

A good connection flow can support:

1. try discovery
2. if successful, show remote models
3. if unsupported or unavailable, allow manual model ID
4. validate on first real request

This is especially important for OpenAI-compatible gateways and private deployments.

Distinguish unsupported from temporarily failed

These cases deserve different UX:

404 /models
401 /models
429 /models
network timeout
valid empty list
malformatted response

Possible interpretations:

Result	Likely meaning
401/403	Credential or permission problem
404	Discovery endpoint may not exist
429	Retry later; do not erase cached catalog
timeout	Network problem; stale cache may still be useful
empty list	Valid but no accessible models, or provider-specific semantics
malformed	Compatibility problem

Do not collapse all of them into “No models found.”

Cache with a stale-while-refresh mindset

Model catalogs usually do not need to block the UI every time a picker opens.

A practical strategy:

open picker
→ show cached models immediately
→ refresh in background
→ merge/replace when fresh result arrives

Store:

discovery timestamp;
source connection;
normalized models;
maybe a provider ETag if supported.

Avoid extremely long cache lifetimes for fast-moving cloud providers.

For local servers, manual refresh may be more important than time-based refresh.

Preserve selected models even when discovery changes

Suppose a chat is configured with model-x and the refreshed catalog no longer contains it.

Do not silently replace the model.

The model may be:

temporarily hidden;
unavailable to this credential;
retired;
renamed;
omitted by a flaky discovery endpoint.

Represent it as unavailable/stale and let the user choose what to do.

That protects conversation reproducibility.

Filtering belongs after discovery

The provider catalog may include models irrelevant to your app:

embeddings;
image-only generation;
moderation;
speech;
rerankers;
deprecated endpoints.

Normalize first, then filter according to the feature surface.

For a chat picker, you might require:

supports text generation
AND supports the selected endpoint family

If the app also supports images/audio, expose those through capability-aware UI instead of hiding them globally.

Capability metadata has multiple sources

A robust system can combine:

provider-declared metadata;
built-in curated metadata;
endpoint-family defaults;
safe capability probes;
user overrides for custom servers.

These sources can disagree.

Give them precedence rules rather than randomly overwriting fields.

For example:

explicit provider metadata
> user-confirmed override
> curated known-model metadata
> endpoint heuristic
> unknown

Unknown is a useful state

Boolean capability fields force bad guesses.

Instead of:

supportsTools: false

consider:

tools: "supported" | "unsupported" | "unknown"

Why?

A model missing from a curated database is not proof that it lacks tools.

This matters especially for new models and private deployments.

Do not infer too much from names

Heuristics such as:

contains "vision" → image input
contains "reasoner" → reasoning
contains "32k" → 32k context

age badly.

They may be useful only as presentation hints, never as strong capability guarantees.

Prefer metadata or explicit user configuration.

Discovery must be connection-aware

A multi-account app can have:

Anthropic / Personal
Anthropic / Work
Custom / Local Mac
Custom / Home Server
OpenRouter / Main

The same model ID can appear under different connections with different:

permissions;
routing;
privacy settings;
limits;
endpoints;
latency.

The selected model should therefore reference a connection plus model ID, not only a global string.

Normalize display names separately

Provider model IDs are often ugly but stable enough for API use.

The UI can display a friendly label while preserving the exact ID:

{
  modelID: "provider/exact-model-slug",
  displayName: "Example Model Pro"
}

Never send the display name to the API unless it is also the canonical ID.

Sorting should reflect user intent

Useful sort modes include:

provider order;
alphabetical;
favorites;
recently used;
capability match;
manually pinned models.

Be careful with dynamic metrics such as latency or price. Those values can become stale quickly and may differ by region or provider route.

If you show them, show source and freshness.

Search should include IDs and labels

Users often paste model IDs from documentation.

Search should match:

display name
exact model ID
provider/author prefix
aliases

But the result should still show the canonical ID somewhere for debugging.

Validate before persisting destructive changes

If refresh returns an empty catalog because of a temporary auth or network problem, do not erase:

favorites;
recent models;
chat selections;
project defaults.

Persist model references independently from the latest discovery cache.

Test discovery as a protocol feature

Useful fixtures include:

normal catalog
empty catalog
1000-model catalog
unknown fields
missing optional fields
duplicate IDs
Unicode labels
401
403
404
429 + Retry-After
500
timeout
malformed JSON
slow response
catalog changes between refreshes

Also test a custom endpoint that supports generation but not discovery.

A clean architecture

The chat selection should not depend on the cache remaining unchanged.

Where BYOKchat fits

A BYOK client with multiple provider accounts and custom OpenAI-compatible connections needs discovery to be both dynamic and conservative. It should use provider model APIs when available, preserve exact model IDs, support manual models when discovery is absent, and keep model capabilities separate from the inventory list.

That lets new models appear quickly without turning every new provider quirk into a UI bug.