How to Design a Custom OpenAI-Compatible Provider Connection

Supporting a custom OpenAI-compatible provider is not just adding a text field for a base URL.

A production connection needs a clear contract for:

endpoint normalization;
authentication;
protected custom headers;
model discovery;
manual model IDs;
TLS and local-network policy;
capability overrides;
validation;
diagnostics;
secret handling.

If these concerns are mixed together, custom-provider support becomes one of the easiest places to create security bugs and confusing UX.

Start with a connection object

Treat each configured endpoint as a first-class connection:

interface CustomConnection {
  id: string;
  name: string;
  baseURL: URL;
  credentialRef?: string;
  protectedHeaders: Record<string, SecretReference>;
  publicHeaders: Record<string, string>;
  modelDiscovery: "auto" | "disabled";
  manualModels: string[];
  transportPolicy: "https" | "privateLanHttp";
}

The exact type is less important than the separation.

Do not store raw secrets inside the same plain-text configuration object if your platform has secure credential storage.

Define exactly what `base URL` means

Users paste many forms:

https://api.example.com
https://api.example.com/v1
https://gateway.example.com/openai/v1
http://192.168.1.10:1234/v1

Your app needs one rule.

A practical rule is:

The saved base URL is the prefix immediately before endpoint resources such as /chat/completions and /models.

Then:

base = https://example.com/v1
chat = https://example.com/v1/chat/completions
models = https://example.com/v1/models

Do not silently append another /v1 when the user already supplied one.

Normalize without changing meaning

Safe normalization can include:

trimming whitespace;
requiring an absolute URL;
removing an unnecessary trailing slash;
normalizing an empty path;
rejecting fragments;
rejecting embedded user/password credentials.

Be cautious about rewriting paths.

This is dangerous:

user enters /api/openai/v1
app decides to replace it with /v1

The custom path may be intentional.

Show the resolved endpoints

A small preview prevents many configuration mistakes:

Base URL
https://gateway.example.com/openai/v1

Chat endpoint
https://gateway.example.com/openai/v1/chat/completions

Models endpoint
https://gateway.example.com/openai/v1/models

This is more useful than a generic “Invalid URL” after the first request fails.

Keep credentials out of the URL

Never encourage:

https://user:password@example.com/v1
https://example.com/v1?api_key=secret

Credentials in URLs leak easily into:

logs;
crash reports;
browser history;
analytics;
diagnostics;
copy/paste screenshots.

Prefer an authorization header or a protected custom header.

Support the common Bearer pattern

Many compatible services use:

Authorization: Bearer <api-key>

A connection editor can expose this as the primary credential field.

Internally, store the key in Keychain or an equivalent secure facility and resolve it only when constructing the request.

Do not persist:

{
  "apiKey": "sk-secret..."
}

inside ordinary app preferences or backup data.

Custom headers are sometimes necessary

Gateways and enterprise proxies may require headers such as:

X-Project-ID
X-Workspace
X-Gateway-Key
api-key

Split headers into two categories:

Public headers

Safe to display and export:

X-Client-Version: 1.2

Protected headers

Credentials or sensitive routing data:

X-Gateway-Key: secret

Protected values should use the same secure-storage rules as API keys.

Reject dangerous header overrides

Do not allow arbitrary custom headers to replace transport-critical headers without thought.

Examples to protect or control:

Host
Content-Length
Transfer-Encoding
Connection

Also decide how duplicate Authorization headers behave.

A safe configuration UI can maintain an allow/deny policy instead of blindly accepting every header name.

HTTPS should be the default

For internet endpoints, require HTTPS.

Plain HTTP exposes:

API keys;
prompts;
responses;
tool arguments/results;
model metadata

to anyone able to observe or modify the network path.

A custom-provider feature should not make http://public-host.example feel equivalent to HTTPS.

Private-LAN HTTP is a separate policy

Local inference servers often run on plain HTTP inside a private network.

If your product supports this, make the exception explicit:

Allow HTTP only for private/local network hosts

Then validate the target rather than globally disabling transport security.

Examples of local/private targets can include loopback, link-local, or RFC1918-style private addresses depending on platform policy.

Do not turn “support my Mac on Wi-Fi” into “allow credentials over arbitrary internet HTTP.”

See HTTP vs HTTPS for Local AI Servers.

Device networking matters

On an iPhone:

http://localhost:1234

means the iPhone itself, not the user’s Mac.

A connection editor for local servers should explain this and encourage the actual LAN hostname/IP of the machine hosting the model.

See Why localhost Does Not Work From Your Phone.

Validation should be staged

One “Test Connection” button often tries to prove too much.

Break validation into stages:

1. URL is structurally valid
2. transport policy permits target
3. server is reachable
4. authentication works
5. model discovery works, if supported
6. selected model can generate
7. optional capabilities can be checked later

This gives better errors.

Do not require `/models`

Some compatible servers implement generation but not discovery.

A resilient client should allow:

Discovery unavailable
→ user enters model ID manually
→ client validates during generation

Do not make model-list support a requirement unless your product explicitly depends on it.

Manual models should be durable

If a user adds:

my-private-model

store it independently from the latest discovered catalog.

A refresh should not erase manual entries.

You can display origin:

Discovered
Manual
Previously used

Capability overrides are useful but dangerous

Private gateways may expose capabilities your app cannot discover automatically.

Advanced users may need overrides such as:

Tools: supported
Image input: unsupported
JSON Schema: supported
Reasoning: unknown

Do not default every unknown feature to supported.

An override should be clearly labeled as user-supplied and easy to reset.

Separate connection validation from model validation

A base URL and API key can be valid while one model ID is wrong.

Represent errors precisely:

connection auth failed
models endpoint unsupported
model not found
model capability unsupported
request invalid

This avoids telling users to re-enter a correct API key when only the model changed.

Redact diagnostics by construction

A useful request diagnostic might show:

POST https://192.168.1.10:1234/v1/chat/completions
HTTP 404
Duration: 120 ms
Request ID: abc123

It should not show:

Authorization: Bearer secret
X-Gateway-Key: secret
full prompt
full tool result

Build redaction before logging, not as a cleanup pass after secrets have already reached the log system.

Never “test” by sending the user’s real conversation

Connection validation should use a tiny synthetic request when generation is necessary:

user: Reply with OK.

Do not upload the current chat merely to validate a new endpoint.

That keeps validation cheap and privacy-preserving.

Handle redirects carefully

HTTP redirects can create credential-leak risks if authorization headers are forwarded to a different host.

A conservative client should:

inspect redirect host changes;
avoid blindly forwarding sensitive headers cross-origin;
surface unexpected redirects during connection testing.

For custom endpoints, silent redirect behavior can hide a misconfiguration.

Timeouts need separate meanings

A connection test should not wait as long as a full reasoning generation.

Consider distinct budgets for:

DNS/connect/TLS
models discovery
small validation generation
normal generation

A slow model is not necessarily an unreachable server.

Store connection identity separately from endpoint text

Chats should reference a stable connection ID:

chat.connectionID = UUID

not a copy of the base URL and API key.

Then users can update a server hostname or rotate a credential without rewriting every conversation record.

Backups should exclude secrets

A provider-configuration export can safely include:

{
  "name": "Home AI",
  "baseURL": "http://192.168.1.50:1234/v1",
  "manualModels": ["local-model"]
}

but should not include the API key or protected header values by default.

On restore, mark the connection as needing credentials.

Connection deletion should revoke local secret references

Deleting a connection should clean up:

Keychain items;
protected custom headers;
transient auth caches;
model discovery cache.

Do not necessarily delete conversations that used the connection. Historical chats should remain readable even when the provider configuration disappears.

Test the configuration layer itself

Useful tests include:

base URL with /v1
base URL without /v1
nested gateway path
trailing slash
Unicode hostname
IPv4 private address
IPv6 literal
localhost
HTTP public host rejected
HTTP private host allowed only under explicit policy
401 models endpoint
404 models endpoint + manual model
redirect to different host
invalid certificate
protected header redaction
credential rotation
connection rename

A provider simulator is ideal for deterministic cases.

A good connection flow

The flow should preserve progress when one optional feature fails.

Where BYOKchat fits

A multi-provider BYOK client benefits from treating custom OpenAI-compatible endpoints exactly like other saved connections: stable identity, isolated credentials, explicit network policy, dynamic or manual models, and adapter-owned capability handling.

That supports cloud gateways and private local servers without turning custom connectivity into a security bypass.