A Reference Architecture for a Production BYOK AI Client

A production BYOK AI client is not a thin wrapper around one chat endpoint.

The moment it supports several providers, local models, files, reasoning, tools, MCP, projects, analytics, backups, and failure recovery, the architecture becomes a real application platform.

The easiest way to keep it understandable is to preserve a few strong boundaries:

provider-native protocols stay in adapters
semantic conversation state belongs to the app
credentials stay in secure storage
context building is separate from persistence
model output never grants authority
reliability operates on logical requests, not random retries
analytics does not require conversation content
provider-specific runtime state never becomes the only copy of user data

This article puts those pieces together into one reference architecture.

The high-level system

The exact modules can be packages, actors, services, or folders. The important part is ownership.

Layer 1: provider-neutral domain model

The core application should understand concepts such as:

conversation
message
content part
attachment
generation
tool call
tool result
usage
project
provider connection
model capability

It should not require every screen to understand native provider payloads.

A semantic message model might be:

interface MessageNode {
  id: string
  parentId: string | null
  role: "user" | "assistant" | "tool"
  parts: ContentPart[]
  createdAt: string
}

Provider adapters translate this into native APIs.

See How to Build a Provider-Neutral AI Message Model.

Do not flatten meaningful provider differences

Provider-neutral does not mean:

pretend all APIs are identical

The core model should preserve common semantics while allowing explicit extensions for:

reasoning state
provider-hosted tools
file references
native finish reasons
provider response IDs
native usage fields

The adapter boundary is where those differences belong.

Layer 2: connection identity

A provider type is not enough.

A user can have:

two Anthropic keys
three OpenAI-compatible endpoints
one OpenRouter account
one local Ollama server
one local LM Studio server

Represent a connection as a first-class object:

interface ProviderConnection {
  id: string
  kind: ProviderKind
  displayName: string
  baseURL?: string
  credentialRef?: string
  configuration: ProviderConfiguration
}

Every generation should record the exact connection used.

Credentials stay outside the ordinary database

The application database can store:

connection ID
provider type
base URL
credential-present flag
safe metadata

The secret value belongs in a platform secure store such as Keychain.

Backups and conversation exports should exclude it.

See How Mobile Apps Should Store AI API Keys.

Protected custom headers are secrets too

If a custom endpoint needs:

Authorization
X-API-Key
X-Internal-Token

treat those values like credentials.

Do not serialize them into:

conversation exports
analytics
logs
crash metadata
project files

Layer 3: provider adapters

A provider adapter owns native request/response semantics.

Conceptually:

interface ProviderAdapter {
  discoverModels() async throws -> [DiscoveredModel]
  generate(request: NormalizedRequest) -> AsyncThrowingStream<ProviderEvent>
  cancel(operation: OperationIdentity) async
}

Not every adapter must implement every capability.

Capability metadata tells the rest of the application what is valid.

Adapters should normalize events, not whole application behavior

The adapter can emit semantic events such as:

response_started
reasoning_delta
text_delta
tool_call_delta
tool_call_completed
usage
response_completed
response_failed

Then the application streaming pipeline can be shared.

Provider-specific event names stay inside the adapter.

See How AI Provider Adapters Work.

Layer 4: capability model

Before sending a request, the app should know or estimate whether the target supports:

text
images
files
tools
structured output
reasoning control
streaming
large context
provider-hosted capabilities

Use three-state knowledge where necessary:

supported
unsupported
unknown

Unknown is especially important for custom compatible endpoints.

See Capability Detection in Multi-Model AI Apps.

Capability validation belongs before network send

The request builder should reject locally when possible:

image attached but model is text-only
tools enabled but model lacks tools
reasoning setting unsupported
context estimate exceeds known capability
structured output requested but unavailable

This produces better UX and avoids wasting API calls.

Layer 5: local conversation persistence

The application should own the durable semantic conversation record.

That enables:

provider switching
branching
search
backup
export
analytics
interrupted-generation recovery

without depending entirely on one provider’s server-side thread object.

Use a graph for branching conversations

A linear UI can sit on top of a message graph:

parentId
currentLeafId

Regenerations create siblings; edits create new downstream branches.

See How AI Chat Branching and Regeneration Should Work.

Separate message state from generation state

A generation record can store:

provider connection
model
settings snapshot
status
usage
timing
provider runtime IDs

A message record stores the semantic conversation content.

This separation makes retries and branches explainable.

Generation status should be explicit

Useful states include:

queued
preparing
streaming
waiting_for_tool
waiting_for_user
completed
interrupted
failed
cancelled

Do not infer all lifecycle state from whether an assistant content string is empty.

Layer 6: context builder

Persistence answers:

What has happened in the conversation?

Context building answers:

What should this specific request send to this specific model now?

Those are different jobs.

The context builder can combine:

project instructions
selected conversation branch
summaries
retrieved file chunks
attachment representations
tool schemas
output reserve
provider-native continuation state

See How to Design Context Management for Long AI Conversations.

Context should be request-specific

Do not persist one globally “final prompt.”

Different targets may require different serialization and budgeting.

For example:

Provider A uses full replay
Provider B uses server continuation
Local model has smaller context

The semantic conversation stays the same; the request context differs.

Context budgeting is a deterministic application concern

The app should reserve space for:

new user content
system/project instructions
tool schemas
attachments/retrieval
expected output

Then trim/summarize/select history according to explicit policy.

Do not wait for random provider context errors as the normal control mechanism.

Layer 7: attachment and retrieval subsystem

Files need provider-neutral local identity.

The attachment layer owns:

local import
metadata
previews
provider upload representations
extraction
OCR
retrieval indexing
deletion

See How File Attachments Flow Through AI APIs.

Retrieval should be separate from ordinary message history

RAG is not simply “append database results to the prompt.”

A retrieval pipeline has:

corpus authorization
query construction
retrieval
reranking
context assembly
provenance

Keep retrieved chunks identifiable as external context.

See RAG Explained.

Layer 8: generation orchestrator

The orchestrator coordinates one logical generation:

It should not contain provider-specific JSON parsing.

Logical generation identity survives attempts

One user action can create several provider attempts due to:

retry
fallback
reconnection
background recovery

Track:

logicalGenerationId
attemptId
providerConnectionId

This keeps analytics and idempotency understandable.

Layer 9: streaming pipeline

Transport bytes should pass through explicit stages:

network bytes
-> SSE/stream framing
-> native provider event
-> normalized provider event
-> generation state reducer
-> renderer state

Do not render directly from socket chunks.

See How AI Streaming Works.

Keep partial output durable enough for recovery

A long generation can fail after useful text has streamed.

Persist or checkpoint partial state so the app can show:

Interrupted response

instead of losing everything.

See How to Resume or Recover an Interrupted AI Generation.

Markdown rendering should tolerate incomplete syntax

Streaming text can end temporarily at:

unfinished code fence $unfinished math **unfinished emphasis

Use an incremental rendering strategy that avoids showing raw Markdown then completely flipping after completion.

See [How to Render Markdown While AI Is Still Streaming](/blog/render-markdown-while-ai-is-streaming/).

## Layer 10: reasoning support

Reasoning is not one universal text field.

Model separately:

```text
reasoning configuration
reasoning summaries/deltas exposed by provider
opaque continuation state
answer content

Do not leak provider-specific reasoning objects into the ordinary assistant text model.

See How Streaming Reasoning Differs From Streaming Answers.

Reasoning continuation state belongs to provider runtime metadata

If a provider needs opaque state to continue reasoning across turns, store it on the generation/path that created it.

The semantic conversation must still be able to fall back to portable messages when switching providers.

See How to Preserve Reasoning Across AI Turns.

Layer 11: tool runtime

The model can propose a tool call.

The host application owns:

schema validation
authorization
approval
execution
idempotency
timeouts
result normalization

The model is not the security boundary.

See How AI Tool Calling Works.

Validate tool arguments before side effects

Pipeline:

parse
schema validate
normalize
authorize
show approval if required
execute
persist result

Never execute arbitrary model JSON directly.

See How to Validate AI Tool Arguments Safely.

Tool approvals bind to exact operations

A meaningful approval should show:

tool
normalized arguments
side effect
server/account

Then authorization is tied to that operation identity.

If arguments change, ask again.

Idempotency is mandatory for retried side effects

If a tool sends email, creates an issue, or modifies a file, retries can duplicate real-world actions.

Use stable operation IDs and downstream idempotency where possible.

See Idempotency for AI Tool Execution.

Layer 12: MCP client runtime

MCP adds remote tool discovery and interactive protocol workflows.

Keep MCP beneath the same application tool boundary:

MCP server -> discovered tool metadata -> app tool catalog -> chat tool exposure -> model

The model should not receive unchecked authority merely because a server advertises a tool.

MCP permissions are host policy

Per-tool policy can be:

Ask
Always Allow
Disabled

or another clear model.

Store it outside prompt text.

See How to Build an MCP Client Permission System.

Remote MCP authorization is separate from tool approval

OAuth answers:

May this client access this protected server/resource?

Tool approval answers:

May this specific operation run now?

Do not conflate them.

See OAuth for MCP Explained.

Interactive MCP workflows need durable operation state

Multi-round input requests and long-running tasks can outlive one model event.

Persist:

operation ID
server identity
request state/task ID
pending user input
subscription/poll state

so app relaunch does not lose the workflow.

Layer 13: request scheduler

A production client benefits from one shared scheduler rather than each screen directly firing network calls.

The scheduler owns:

concurrency
priority
queueing
cancellation
not-before timestamps
provider/connection fairness
background maintenance

See How to Build an AI Request Queue.

Queue logical work, not raw retries

The queue should know that:

attempt 2

belongs to the same logical request as attempt 1.

This prevents retries from bypassing normal concurrency limits.

Layer 14: reliability policy

The reliability layer classifies failures and decides:

retry
wait for rate limit
fail fast
reconcile unknown outcome
fallback if allowed
open circuit

Provider adapters supply native error metadata; shared policy uses normalized categories.

See How to Classify AI API Errors.

Timeouts should be phase-specific

Separate:

connect timeout
first response timeout
first text timeout
idle stream timeout
overall deadline
tool timeout
background operation deadline

A single 30-second timeout cannot describe all AI request phases well.

See Timeout Design for AI Applications.

Rate-limit handling belongs to scheduling

If a response says:

retry after T

the scheduler can set:

connection.notBefore = T

rather than sleeping inside a worker while blocking other work.

Circuit breakers protect unhealthy targets

Repeated transient failures can temporarily stop dispatch to a target.

Do not count:

bad API key
context overflow
unsupported feature
user cancellation

as provider outage evidence.

See Circuit Breakers for AI Providers.

Fallback never overrides privacy policy

If a user selected:

local only

then local failure does not authorize cloud fallback.

Provider switching changes the data destination and sometimes capabilities.

See Cloud Model vs Local Model Routing.

Layer 15: local-first analytics

The client can measure:

input/output tokens
estimated spend
request count
TTFT
duration
generation speed
reliability
tool calls
provider/model breakdown

without collecting prompt/response content.

See Privacy-Preserving Analytics for AI Apps.

Analytics should be event-schema driven

Define events such as:

{
  "event": "generation_completed",
  "provider_kind": "anthropic",
  "model_id": "...",
  "input_tokens": 1234,
  "output_tokens": 456,
  "ttft_ms": 820,
  "duration_ms": 9400
}

Exclude:

prompt text
response text
reasoning text
API keys
tool arguments/results
attachment contents
private URLs/hostnames

Cost accounting keeps usage and pricing separate

Store usage per generation, then derive estimated cost using a versioned pricing catalog.

Do not store only one opaque dollar total.

See How to Estimate Per-Conversation AI Cost.

Layer 16: search

Search is a derived local index over semantic conversation data.

Start with lexical search and add semantic retrieval only when the privacy boundary is explicit.

See How to Design AI Conversation Search.

Search indexes are rebuildable state

On deletion/restore/schema migration:

primary data -> rebuild index

Do not treat an old vector/full-text index as the canonical user record.

Layer 17: projects/workspaces

Projects compose reusable defaults:

instructions
files
provider/model
tools
retrieval corpus
settings

but historical generations still preserve the effective configuration used at the time.

See How to Build Reusable AI Projects and Workspaces.

Defaults and security constraints are different

Project model default:

can be overridden by chat

Project local-only privacy policy:

should act as a hard constraint unless explicitly changed

Do not merge them with one generic dictionary precedence function.

Layer 18: backup and restore

A local-first client needs versioned backups containing user-owned data:

conversations
projects
attachments
provider configuration metadata
tool history
library state

while excluding credentials.

See Backup and Restore Security for Local AI Chats.

Restore treats archives as untrusted

Validate:

schema version
object counts
IDs/references
paths
attachment sizes
archive traversal

before committing local mutations.

Layer 19: portable export

Backup optimizes for app restoration.

Portable export optimizes for human/machine interoperability.

Support formats such as:

Markdown selected path
JSON semantic graph
archive + attachments

See How to Export AI Conversations Portably.

Layer 20: deletion

Deletion should traverse the data graph:

conversation rows
attachments
OCR/extracted text
search index
embeddings
credentials when requested
queued work
provider-owned resources when explicitly modeled

See How to Delete AI App Data Correctly.

Layer 21: app lifecycle and recovery

Mobile/desktop apps are suspended, terminated, relaunched, and moved across networks.

Treat lifecycle interruption as normal.

Persist enough state to recover:

streaming generation status
background provider job IDs
queued retries
pending tool approval
MCP tasks
file uploads

See How to Handle App Backgrounding During AI Generation.

Do not assume an interactive stream survives backgrounding

The UI should be able to reopen a conversation and reconcile:

completed
interrupted
still running remotely
unknown

instead of requiring the original in-memory task object to survive forever.

Long-running provider jobs need durable local operation IDs

Store both:

app operation ID
provider job/response ID

Then retries and webhooks/polling can refer to one logical operation.

See How Long-Running AI Tasks Work.

Layer 22: offline and degraded mode

Capabilities can fail independently:

cloud provider offline
local model healthy
MCP offline
web search unavailable
file service unavailable

Preserve what still works and tell the user what changed.

See Designing Offline and Degraded Modes for AI Apps.

Never weaken security to recover availability

Do not respond to an outage by:

disabling TLS validation
silently changing HTTPS to HTTP
sending prompts to another provider
forwarding credentials across redirects

Availability is not authority.

Layer 23: security boundaries

Threat-model the client around assets:

API credentials
conversation data
attachments
tool authority
provider connections
local files
OAuth tokens

Then identify trust boundaries between:

UI
local persistence
provider APIs
MCP servers
tools
web content
custom endpoints

See Threat Modeling a BYOK AI Client.

Prompt injection is an input problem, not an authorization model

Tool results, web pages, files, and retrieved documents can all contain adversarial instructions.

The host still enforces:

which tool is exposed
which arguments are valid
which action needs approval
which resource scope is allowed

See How to Prevent Prompt Injection From Tool Results.

Custom endpoints need strict URL/TLS handling

Normalize base URLs and apply:

allowed schemes
TLS validation
redirect rules
credential forwarding restrictions
private-network policy

Do not concatenate strings blindly.

See Certificate Validation for Custom AI Endpoints.

Layer 24: observability

Production debugging requires IDs across layers:

conversation ID
generation ID
logical request ID
attempt ID
provider request ID
tool execution ID
background operation ID

Then logs can correlate state without including prompt content.

Timing should identify phases

Useful timestamps:

queued
network started
headers/first event
first text
last text
completed
persisted

This separates network, provider, model, tool, and local persistence latency.

See Observability for Streaming AI Requests.

Layer 25: testing architecture

A provider-neutral core needs deterministic tests for:

success
streaming
malformed events
429
503
cancellation
timeouts
tool loops
side effects
attachments
background jobs

Do not depend on real provider outages.

See How to Build an AI Provider Simulator for Testing.

Test adapters with native fixtures

Each provider adapter should have contract fixtures for:

request JSON
headers
native stream events
finish states
usage
error mapping

Shared application tests then consume normalized events.

Test the state reducers separately from networking

Given events:

response_started
text_delta("Hel")
text_delta("lo")
usage(...)
response_completed

assert deterministic final generation state.

This makes UI behavior easier to test than only launching full network scenarios.

Use failure injection across boundaries

Important crash points:

after upload success before local persistence
after tool side effect before result persistence
after provider job creation before ID save
after stream completion before message finalize
after backup parse before transaction commit

These are where duplicate actions and lost state hide.

Keep UI state derived from durable application state

A screen should render:

generation.status
partial text
pending approval
provider state

rather than owning the authoritative network task lifetime itself.

This makes navigation and relaunch safer.

Model and reasoning controls sit above capabilities

The settings UI asks the capability layer what is valid, then the request builder maps semantic settings to provider-native fields.

See How to Design Model and Reasoning Controls Without Confusing Users.

Model comparison is an analytics + capability view

The client can compare:

features
context
recent latency
recent reliability
estimated cost
user preference

without one universal quality score.

See How to Compare AI Models Inside a Client App.

Suggested package/module boundaries

One practical organization:

Domain/
  Conversation
  Message
  Generation
  Attachment
  Project

Providers/
  Core
  Anthropic
  Gemini
  OpenAICompatible

Execution/
  ContextBuilder
  GenerationOrchestrator
  RequestScheduler
  Reliability

Tools/
  ToolCatalog
  ToolRuntime
  MCPClient
  Permissions

Data/
  ConversationStore
  SecureCredentialStore
  AttachmentStore
  SearchIndex
  AnalyticsStore

Portability/
  Backup
  Restore
  Export

Testing/
  ProviderSimulator
  Fixtures

The exact names are less important than preventing cycles.

Dependency direction should point inward

A healthy dependency graph looks roughly like:

UI -> Application services -> Domain
Provider adapters -> Provider core/domain protocols
Persistence implementations -> storage protocols

The domain model should not import a concrete HTTP client or UI framework.

Avoid one giant singleton AIService

A common early architecture becomes:

AIService
- talks to all providers
- stores chats
- renders streams
- runs tools
- writes analytics
- reads keychain
- handles backups

This becomes impossible to test or evolve safely.

Split by responsibility before provider count and feature count make it painful.

Avoid provider switches scattered through the app

Bad pattern:

switch provider {
  case anthropic: ...
  case gemini: ...
  case openai: ...
}

inside:

chat view
settings
analytics
tool loop
file upload

Keep provider dispatch behind adapter protocols/capabilities.

Avoid raw provider JSON in persistence

Raw payloads are useful diagnostics but brittle as the only stored conversation format.

Persist semantic content and optional native extensions.

This lets the app evolve provider APIs without migrating every historical message payload.

Avoid hiding critical semantics in UI-only state

Examples:

selected branch only in view memory
pending tool approval only in modal state
retry count only in progress indicator
provider job ID only in task closure

If recovery depends on it, make it durable application state.

Avoid automatic fallback that changes data destination silently

A production architecture should route through explicit policy:

primary target
allowed fallback targets
privacy constraints
capability constraints

The router cannot invent permission to send data elsewhere.

Avoid one global provider health number

Health is scoped to:

connection
endpoint
model/operation when necessary
recent time window

Authentication errors and capability mismatches are not global service outages.

See Health Scoring AI Providers Without Fake Precision.

Avoid telemetry that becomes a shadow chat database

You do not need:

prompt
response
reasoning
tool payload
attachment text

for basic product reliability analytics.

Store only what each metric requires.

Avoid coupling export to current UI layout

If export is implemented by scraping rendered views, it becomes fragile.

Export from semantic conversation data.

UI and export are both renderers of the same domain model.

A complete request lifecycle

Reliability policy wraps each external operation.

A complete data ownership model

User-owned durable data:
- conversations
- projects
- attachments
- selected settings
- backups/exports

Secure secrets:
- API keys
- OAuth tokens
- protected headers

Derived/rebuildable data:
- search index
- embeddings when locally generated
- thumbnails
- caches

Provider runtime state:
- response IDs
- provider file IDs
- background job IDs
- opaque reasoning state

Deletion and backup policies should follow these categories.

Useful system-wide invariants

A mature BYOK client can assert:

no provider credential appears in conversation storage
no secret appears in analytics or normal logs
provider-native state never replaces semantic conversation history
unsupported model capabilities are rejected before send when known
only selected branch enters context
new tool execution always passes validation/authorization
retry does not duplicate side effects when idempotency guarantees exist
fallback never weakens explicit privacy policy
unknown provider outcome is reconciled before unsafe replay
partial generations remain recoverable
backups exclude credentials
exports are provider-neutral at the core
search/deletion derive from primary data lifecycle

These invariants are more valuable than any particular framework choice.

A staged implementation order

If building from scratch, a practical order is:

1. semantic conversation model
2. one provider adapter
3. streaming state reducer
4. secure connection/credential model
5. persistence + recovery
6. capability layer
7. second provider to validate abstraction
8. context budgeting
9. attachments
10. tools
11. scheduler/retry/rate limit
12. MCP
13. analytics
14. projects/search
15. backup/export
16. simulator/failure injection

Do not design 20 provider abstractions before proving the model with two genuinely different APIs.

Optimize boundaries before micro-performance

Architecture problems are more expensive than a few extra allocations.

Once boundaries are correct, measure:

initial load
stream rendering
search indexing
large conversation persistence
attachment memory
local inference pressure

and optimize with evidence.

Performance should preserve correctness

Examples:

batch partial-text persistence
reuse network connections
incrementally render Markdown
cache model capability metadata
rebuild derived indexes asynchronously

But never optimize by:

skipping security validation
keeping secrets in convenient plaintext caches
losing operation identity
merging unrelated retry layers

The architecture should make the user model simple

Internally, the system can be sophisticated.

The user should still understand:

I choose my provider/model.
My API key is stored securely.
My chat is local unless I send it to a provider/tool.
I can see which tools are enabled.
I can stop, retry, export, and delete my data.
If something fails, the app tells me what happened instead of hiding it.

That is a strong BYOK product contract.

Final checklist

Provider boundary

• Native APIs live behind adapters.
• Model capabilities are explicit.
• Custom compatible endpoints can express unknown capability.
• Connection identity includes account/endpoint, not just provider type.

Data

• Semantic conversation history is app-owned.
• Branches are preserved.
• Attachments have provider-neutral identity.
• Provider runtime IDs are supplemental metadata.

Security

• Credentials use secure storage.
• Protected headers follow secret rules.
• TLS/URL handling is strict.
• Prompt/tool content never grants authority.

Execution

• Context building is separate from persistence.
• Streaming uses normalized semantic events.
• Tool calls pass validation, authorization, and idempotency rules.
• MCP uses the same host permission boundary.

Reliability

• Logical requests and provider attempts have separate IDs.
• Timeouts are phase-specific.
• Rate limits feed scheduling.
• Circuit breakers use scoped transient evidence.
• Fallback respects privacy/capability policy.

Lifecycle

• Partial generations can recover.
• Background jobs persist IDs.
• App relaunch reconciles durable operations.
• Offline mode preserves available local capabilities.

Portability

• Backups are versioned and secret-free.
• Exports have provider-neutral core formats.
• Deletion covers derived state.
• Search indexes are rebuildable.

Testing

• Provider adapters have native fixtures.
• Simulator can inject deterministic failures.
• Side-effect crash boundaries are tested.
• Privacy invariants have sentinel-secret tests.

Where BYOKchat fits

A product like BYOKchat can use this architecture to keep many provider connections, local models, projects, MCP servers, files, tools, analytics, and conversation history inside one coherent local-first system. The important part is not the number of supported features; it is that each feature composes through stable boundaries rather than bypassing them.

That is what lets a BYOK client grow without becoming a collection of provider-specific special cases.

Further reading

• How a Multi-Provider AI Client Is Architected
• Designing a Local-First AI Chat Architecture
• How AI Provider Adapters Work
• How to Test a Multi-Provider AI Client
• How to Build an AI Provider Simulator for Testing