Cloud Model vs Local Model Routing

Once an AI client supports both cloud providers and local servers, a new question appears:

Which model should handle each request?

The wrong answer is a single global rule:

always local

or:

always cloud

Local and cloud execution optimize different things.

A useful router evaluates the actual request requirements and the user’s policy.

Start with hard requirements

Before ranking models, eliminate targets that cannot satisfy the request.

Examples:

image input required
tool calling required
strict JSON Schema required
very large context required
specific provider-hosted tool required
offline execution required
data must not leave device/network

Hard requirements are filters, not preferences.

Then rank preferences

Among valid targets, the user may care about:

privacy;
latency;
cost;
quality/capability;
availability;
battery/power;
reproducibility;
provider independence.

These can be expressed as a policy rather than a mysterious “smart routing” score.

Privacy is not one boolean

A local model keeps model inference on local hardware, but the workflow can still send data remotely through:

remote MCP tools;
web search;
cloud embeddings;
remote file retrieval;
analytics;
synchronization.

A routing policy should evaluate the full workflow.

If the user requires:

no prompt content leaves this device

then a local model plus remote tools still violates the requirement.

Cloud can have capabilities local models lack

A cloud provider may offer:

stronger frontier reasoning;
very large context;
hosted search;
advanced multimodal input/output;
specialized agents;
managed background execution.

If the request needs one of those capabilities, local-first preference should not force an incapable model.

Local can work when cloud cannot

A local model can be valuable when:

internet is unavailable;
provider service is down;
the user does not want to send content externally;
API quota is exhausted;
the task is simple enough for the local model.

Offline/degraded mode should be explicit and truthful about capability differences.

Latency has multiple components

Cloud latency includes:

network + provider queue + prompt processing + generation

Local latency can include:

LAN/loopback + model load + prompt processing + generation

A cold local model can have worse TTFT than cloud even if steady-state generation is fast.

Measure actual request lifecycle rather than assuming “local = instant.”

Cost is also multidimensional

Cloud usage commonly has explicit API cost.

Local inference has no per-request provider bill, but still consumes:

hardware;
electricity;
storage;
battery;
user time;
machine responsiveness.

A router can treat local marginal monetary cost as low without claiming it is literally free.

Context can flip the decision

A local model may run well at short context but become memory-heavy at long context.

A cloud model may support a much larger request comfortably.

Before routing, estimate:

input context
+ tool schemas
+ attachments
+ output reserve

Then filter targets by practical context capability.

Tool compatibility matters

Suppose a chat has enabled tools.

A local target must support the required tool-calling protocol well enough for the workflow.

Do not route to a model that can answer text but cannot produce reliable tool calls if the request depends on tools.

Similarly, provider-hosted tools may force use of that provider.

Reasoning state can reduce portability mid-chat

If the current conversation uses provider-native reasoning state, switching to a local model may require rebuilding context from portable messages rather than continuing opaque provider state.

That is possible only if the application owns a semantic conversation record.

See How to Switch AI Providers Mid-Conversation.

Routing per chat is easier to understand than routing every turn invisibly

Fully automatic per-request routing can surprise users:

Why did this sensitive turn go to cloud?
Why did the answer quality change?
Why is the bill different?

A simpler product can use:

chat default = local/cloud/provider/model

with optional fallback under explicit rules.

Advanced automatic routing can be added later.

Make fallback constraints explicit

Example policy:

Primary: local model
Fallback: cloud only if local unavailable
Never cloud for chats marked Private Local

Another policy:

Primary: cloud model
Fallback: local if provider unavailable
Tools required: disable fallback if local model lacks tools

Fallback should re-run capability and privacy validation every time.

Do not silently violate privacy for availability

If the user says:

local only

then the correct response to local-server failure is:

Local model unavailable

not:

quietly send the prompt to a cloud provider

Availability is not permission.

Use a requirements-first router

This is easier to reason about than one opaque score.

A simple routing type

struct RoutingPolicy {
    var preferredTarget: TargetKind
    var allowFallback: Bool
    var allowCloudFallback: Bool
    var requireLocal: Bool
    var requiredCapabilities: Set<Capability>
}

The exact representation can be richer, but explicit fields make policy review possible.

Health should be measured separately

A local server can be:

reachable but model not loaded
reachable but overloaded
unreachable

A cloud provider can be:

available
rate limited
regionally degraded
credential invalid

Do not convert authentication errors into “provider unhealthy.”

Health scoring should use transient service signals, not configuration mistakes.

Local routing can account for machine state

Advanced local routing may consider:

model already loaded;
current queue depth;
machine on battery;
memory pressure;
user-configured concurrency.

These are local operational signals, not model quality claims.

Keep the rules understandable.

Cloud routing can account for provider constraints

Cloud candidates may differ by:

account permissions;
rate limits;
privacy mode;
routing policy;
model availability.

Again, a model ID alone is not the full target.

Cache routing decisions carefully

A model capability profile can be cached.

A health/latency decision should be much shorter-lived.

Do not persist “provider A is fastest” as a durable truth from one request.

Explain routing in diagnostics

A request record can store a reason such as:

Selected local-model because:
- user preference: local first
- tools not required
- context fits
- local server available

or:

Cloud fallback used because local server was unreachable.

This makes automatic behavior trustworthy and debuggable.

Cost estimates should be target-specific

Before sending, a cloud target may have an API-cost estimate based on current pricing metadata.

A local target can instead show:

Local — no provider API charge

without pretending hardware/power cost is zero.

If pricing metadata is stale, label it as an estimate.

Model comparisons should use actual task needs

Do not create a universal rank such as:

cloud model = 95
local model = 72

Quality is task-dependent.

For routing, use measurable/known capability requirements first, then user preference.

If you add evaluation-based routing, base it on the user’s workload and clearly describe the evaluation method.

Test policy conflicts

Important cases:

local-only + local offline
cloud preferred + no internet
image required + local text-only
MCP tool required + both support tools
strict schema required + only cloud supports it
cloud fallback disabled
local model overloaded
provider rate limited
user switches model mid-stream

The router should produce deterministic, explainable outcomes.

Where BYOKchat fits

A multi-provider BYOK client already has the core ingredients for local/cloud routing: connection identity, model capability profiles, context budgeting, tool requirements, analytics, and user-controlled provider choice. Routing can build on those layers without taking provider choice away from the user.

The strongest default is still explicit selection; automatic fallback/routing should be an opt-in policy with hard privacy boundaries.