How a Multi-Provider AI Client Is Architected

A multi-provider AI client should not be built as one giant switch statement around HTTP requests.

The difficult part is not calling several APIs. The difficult part is creating one coherent product model while providers disagree about messages, streaming, reasoning, tools, files, state, errors, and capabilities.

A durable architecture separates three concerns:

product semantics
provider-neutral execution contracts
provider-native protocol details

The goal is not to make every provider look identical. It is to keep the rest of the application stable while preserving differences that matter.

Start from the product, not the provider SDK

A chat product needs concepts such as:

conversations;
user and assistant turns;
attachments;
model selection;
system/project instructions;
generation state;
tool calls and results;
reasoning presentation;
usage and reliability metadata.

Those concepts should not be defined by whichever provider you integrated first.

A common failure is to make the first provider’s request object the application database schema:

Provider A messages
        ↓
local persistence
        ↓
UI

The moment Provider B has a different content-block model, tool-call shape, or reasoning item, the product becomes a translation problem everywhere.

Instead:

The canonical model belongs to your application. Provider adapters translate at the edge.

Layer 1: canonical conversation state

The persisted conversation should describe what happened semantically.

A simplified model might contain:

type ConversationItem =
  | UserMessage
  | AssistantMessage
  | ToolCall
  | ToolResult
  | AttachmentReference;

Each item can carry provider metadata when required, but portable meaning should remain explicit.

For example:

type AssistantMessage = {
  id: string;
  text: string;
  provider?: ProviderMetadata;
  generation: GenerationMetadata;
};

Do not make the entire item an opaque provider JSON blob.

Why? Because the product may need to:

render history offline;
export it;
search it;
back it up;
edit and branch it;
switch providers;
rebuild context after an API change.

See Stateful vs Stateless AI Conversations.

Layer 2: a generation request expressed in product terms

The execution layer receives an intent such as:

type GenerationRequest = {
  conversationID: string;
  model: ModelSelection;
  instructions: InstructionSet;
  context: ContextBundle;
  enabledTools: ToolDefinition[];
  attachments: AttachmentInput[];
  preferences: GenerationPreferences;
};

This should describe what the application wants, not how Anthropic, Gemini, OpenAI, or a compatible endpoint spells it.

A provider adapter can then answer:

Can I represent this request?
If yes, how?
If partially, what should degrade?
If no, what error should the product show before sending?

Layer 3: provider adapters

Each adapter owns provider-native mechanics:

interface ProviderAdapter {
  capabilities(model: ModelID): ModelCapabilities;
  buildRequest(input: GenerationRequest): NativeRequest;
  stream(request: NativeRequest): AsyncSequence<ProviderEvent>;
  normalizeError(error: unknown): ProviderError;
}

Real interfaces will be more detailed, but the responsibility boundary matters.

The adapter should know:

endpoint paths;
authentication headers;
request schema;
content block conversion;
streaming event types;
tool formats;
reasoning controls/state;
file references;
usage metadata;
provider request IDs;
provider-specific errors.

Shared chat views should not.

Normalize events, not raw bytes

Streaming is one of the easiest places to leak provider logic into the UI.

Provider A might emit text deltas. Provider B might emit content-block start/update/end events. Another might use an OpenAI-compatible SSE shape but add custom fields.

Convert native stream events into a small set of application events:

enum GenerationEvent {
  case started
  case textDelta(String)
  case reasoningDelta(String)
  case toolCallDelta(...)
  case toolCallCompleted(...)
  case usage(...)
  case completed(...)
}

Then the renderer responds to semantic events instead of parsing provider frames.

Do not normalize too aggressively. If a provider exposes a meaningful event the product can use, add an explicit semantic type rather than flattening everything into textDelta.

Capability profiles are central

A multi-provider client needs to know more than a model name.

A capability profile can describe:

type ModelCapabilities = {
  streaming: Bool;
  tools: Bool;
  parallelTools: Bool?;
  reasoning: ReasoningCapability;
  attachments: AttachmentCapabilities;
  structuredOutput: StructuredOutputCapability;
  contextWindow: Int?;
  maxOutput: Int?;
};

Values may come from:

provider documentation;
model-list metadata;
local curated knowledge;
custom-endpoint configuration;
cautious runtime detection.

The UI should use capabilities to decide which controls are valid.

Bad:

if provider == "X" show reasoning toggle

Better:

if selectedModel.capabilities.reasoning != .none

This scales when one provider offers many model families with different features.

Keep model identity scoped to a connection

A model ID is not globally unique.

For example, two OpenAI-compatible servers may both advertise:

model = "llama-3"

but represent completely different deployments.

Use an identity like:

(provider connection ID, model ID)

A provider connection itself may include:

provider kind
account identity
base URL
credential reference
custom headers
private-network policy

This prevents accidental cross-account or cross-endpoint state reuse.

Credentials are connection state, not conversation content

A chat can reference a provider connection, but credentials should not be copied into the conversation database.

Prefer:

conversation → connection ID
connection → secure credential reference
secure store → API key/token

This makes key rotation and account deletion tractable.

It also makes backups safer because the conversation export can omit secrets while still preserving which logical connection was used.

See How to Store API Keys Safely.

Context construction should happen before provider translation

A conversation can contain more data than any one request should send.

The context manager should decide:

which turns to include;
what older content to summarize;
which attachments/evidence matter;
how much output to reserve;
which tools are enabled.

Then the adapter translates that selected context into native request format.

This separation is critical when the user switches to a model with a smaller context window.

See How to Design Context Management for Long AI Conversations.

Tool orchestration belongs above individual adapters

Provider APIs expose tool calling differently, but the application usually wants one tool execution system.

A good boundary is:

provider adapter
→ normalized tool call
→ host authorization policy
→ tool executor / MCP layer
→ normalized tool result
→ provider adapter

The adapter handles protocol shape. The host owns whether a tool may run.

This keeps security policy provider-neutral.

See How AI Tool Calling Works and How MCP Tool Permissions Work.

Reasoning needs a dedicated abstraction

Reasoning-capable models are especially difficult to flatten.

A provider may expose:

no reasoning control;
an effort setting;
visible reasoning summaries;
hidden reasoning state;
encrypted continuation items;
provider-native opaque blocks.

Do not treat all of these as a single reasoningText: String.

A more accurate abstraction can distinguish:

configuration
visible presentation
opaque continuation state
usage/accounting

Only the parts safe and intended for the user should be rendered.

Provider-native state can be stored as scoped metadata when required for continuation.

Errors should become actionable categories

Raw provider errors are useful for diagnostics but terrible as the product’s only error model.

Normalize into categories such as:

authentication
permission
invalid_request
model_unavailable
rate_limited
transient_provider
network
context_too_large
unsupported_capability
cancelled
unknown

Preserve the provider’s request ID, code, and safe message for debugging.

The normalized category drives UX and retry policy; native metadata preserves fidelity.

See AI API Errors 401, 403, and 429 and Designing Reliable AI Retries.

Model discovery needs caching and reconciliation

Some providers expose model-list endpoints. Some compatible servers do too. The returned metadata can be incomplete.

A client can combine:

remote model list
+
locally known capability overrides
+
user-created custom models

Do not delete a user’s model selection immediately because one discovery request temporarily fails.

Persist stable user configuration separately from the latest discovery snapshot.

Persistence should record execution metadata without coupling the schema

Useful generation metadata includes:

provider connection ID
model ID
request timestamps
TTFT
duration
input/output usage
finish reason
provider request ID
retry/fallback history

This powers diagnostics and analytics.

But do not require the entire raw provider response for the conversation to remain readable.

Provider switching should rebuild, not mutate old history

When the user changes providers, existing turns remain what they were.

The next request should:

select portable conversation/context;
omit or translate provider-specific continuation state;
validate the new model’s capabilities;
build a new native request.

Do not rewrite old messages into the new provider’s schema in the database.

The adapter is an execution boundary, not a migration engine for canonical history.

Custom OpenAI-compatible endpoints need first-class treatment

“OpenAI-compatible” endpoints are rarely identical in every detail.

A custom connection may need:

base URL
API key or no auth
custom headers
model ID
HTTPS or explicit private-LAN HTTP policy
capability overrides

Treat compatibility as a protocol family, not proof that every OpenAI feature exists.

See What Is an OpenAI-Compatible API?.

Test the abstraction against disagreement

A weak abstraction looks good when all providers behave similarly.

Test cases should deliberately include differences:

one provider has tools, another does not;
one streams usage, another reports it only at the end;
one exposes reasoning state, another only text;
one supports files by ID, another requires inline data;
one model has a smaller context;
one endpoint returns malformed SSE;
one compatible server ignores an unsupported parameter;
one provider changes model-list metadata.

The architecture is successful when these differences stay localized instead of infecting the entire product.

A reference component map

Each component has a narrow job. That is what keeps additional providers from multiplying complexity everywhere.

Architecture checklist

Before calling a client genuinely multi-provider, verify that:

canonical conversations do not use one provider’s schema as the database model;
provider/model identity is scoped to a connection;
credentials live outside conversation data;
context assembly is provider-neutral;
adapters own request/stream/error translation;
the UI is driven by model capabilities, not provider-name conditionals;
tool authorization is host-controlled;
provider-specific reasoning state stays scoped;
custom compatible endpoints can override capabilities;
errors preserve both normalized categories and native diagnostics;
provider switching does not require rewriting history;
deterministic simulators test divergent provider behavior.

Where BYOKchat fits

BYOKchat uses the multi-provider pattern because one local conversation system must support Anthropic, Gemini, DeepSeek, OpenRouter, NVIDIA, custom OpenAI-compatible services, and private-network endpoints while also supporting tools, reasoning, attachments, projects, analytics, and recovery.

The architectural goal is not to hide which provider the user selected. It is to keep provider choice explicit without making every feature depend on provider-specific code.