Certificate Validation for Custom AI Endpoints

Supporting custom AI endpoints means supporting custom trust decisions.

A fixed provider integration can usually assume:

https://api.vendor.example

with ordinary public certificate trust.

A custom endpoint may instead be:

https://ai.company.example
https://192.168.1.20:8443
https://llm.home.arpa
http://192.168.1.20:1234

That creates a much larger networking and security surface.

The safest default is simple:

Use normal platform TLS trust evaluation for HTTPS, reject invalid certificates, and keep exceptions narrow and user-controlled.

TLS does more than encrypt traffic

A secure TLS connection aims to provide:

confidentiality;
integrity;
server authentication.

Encryption without authenticating the server is not enough.

If an app accepts any certificate, an attacker who can intercept traffic can present their own certificate and read or modify the session.

The platform trust store should be the default

For ordinary HTTPS endpoints, rely on the operating system’s standard trust evaluation.

That verifies properties such as:

certificate chain
trusted root
validity period
hostname match
signature validity
revocation-related policy where applicable

Do not replace a mature system trust implementation with ad hoc certificate parsing unless the product genuinely needs custom trust behavior.

App Transport Security adds another layer on Apple platforms

For URL Loading System networking such as URLSession, App Transport Security encourages secure HTTPS configurations and blocks connections that do not meet its requirements unless the app declares exceptions.

The important architectural point is:

secure transport is the baseline
exceptions are explicit configuration

Do not add a broad ATS disable simply because one local-development endpoint uses HTTP.

Self-signed certificates are not inherently unencrypted

A self-signed HTTPS endpoint can still encrypt traffic.

The problem is trust.

The client cannot automatically know whether:

self-signed cert A

belongs to the intended server or an attacker.

If the app chooses to support self-signed certificates, it needs an explicit trust model.

Possible approaches include:

install a trusted private CA on the device;
pin a specific certificate or public key;
ask the user to verify a fingerprint out of band;
reject self-signed certificates entirely.

Installing a private CA is often cleaner than per-app bypasses

In managed or advanced environments, a private CA can issue certificates for internal AI services.

Then the device trust store knows that CA and normal hostname validation still applies.

This is usually easier to reason about than code that says:

if host is custom, trust any certificate

which destroys server authentication.

Never implement “accept all certificates” as a user convenience

A toggle such as:

Ignore SSL errors

is extremely broad.

It can turn:

certificate expired
hostname mismatch
unknown issuer
intercepting proxy
malicious certificate

into the same silent success.

If a development-only bypass exists, keep it out of normal production flows and never attach real credentials to it.

Certificate pinning is powerful but expensive to operate

Pinning constrains acceptable server identity beyond normal public trust.

You might pin:

a certificate;
a public key;
a set of backup keys.

Benefits:

narrows trust for a known endpoint;
can defend against some CA compromise scenarios.

Costs:

certificate/key rotation can break clients;
recovery paths are harder;
custom endpoints become less flexible;
pin update mistakes can create outages.

Pinning is usually a poor default for arbitrary user-configured AI servers because the app does not control their certificate lifecycle.

If you support pinning, bind it to one connection identity

A pin should belong to:

connection + host + trust policy

not to every request globally.

For example:

{
  "connection_id": "A1",
  "host": "llm.example.internal",
  "trust_mode": "pinned_spki",
  "pin": "sha256/..."
}

Do not let one custom trust choice weaken unrelated provider connections.

Hostname validation matters

A certificate can be cryptographically valid yet belong to another host.

If the user configures:

https://ai.example.com

and the server presents a certificate for:

other.example.com

that should fail normal trust evaluation.

Do not disable hostname validation just because the chain is otherwise trusted.

IP-address endpoints are awkward for HTTPS

Local users often enter:

https://192.168.1.20:8443

The certificate must be valid for that identity under the platform’s hostname/IP validation rules.

A certificate issued only for:

llm.home.example

will not necessarily validate when the client connects by raw IP.

A better local setup may use a stable hostname plus a trusted certificate or private CA.

Plain HTTP should be a separate transport mode

Do not model HTTP as “HTTPS with certificate checks disabled.”

They are different security states.

For a BYOK client, a reasonable policy is:

HTTPS -> normal trust evaluation
private-network HTTP -> explicit opt-in for local servers
public HTTP -> reject

This is easier to audit than many mixed exception flags.

See HTTP vs HTTPS for Local AI Servers.

Private-LAN HTTP still has risks

A request over plain HTTP can be read or modified by anyone with sufficient access to that network path.

That matters even when the model server itself has no API key.

Traffic may contain:

prompts;
documents;
generated answers;
tool schemas;
tool results.

Make the insecure transport state visible to the user.

Do not send cloud credentials to insecure local endpoints

A custom connection might combine:

http://192.168.1.20:8080
Authorization: Bearer SECRET

That credential will travel in plaintext on the network.

If the app permits such a combination, it should be an explicit high-risk configuration rather than an accidental outcome of generic custom headers.

Prefer local server auth schemes designed for the actual threat model, or HTTPS.

Redirects cross a trust boundary

Suppose:

https://trusted.example/v1

redirects to:

https://other.example/v1

The new destination has a different server identity.

The client should reconsider:

whether the redirect is allowed;
whether authorization headers should follow;
whether custom pins apply;
whether the new host is still inside user policy.

Do not treat redirects as transparent string rewrites.

TLS failures should remain distinct

Do not collapse all TLS errors into:

Cannot connect

Useful categories include:

certificate expired
certificate not yet valid
hostname mismatch
untrusted issuer
pin mismatch
TLS protocol failure
connection refused
DNS failure

The UI can stay concise while diagnostics retain the category.

Avoid teaching users to bypass security blindly

A bad error flow:

SSL error. Disable certificate validation?
[Disable]

A better flow explains:

The certificate for this endpoint is not trusted.
Verify the server address and certificate configuration.

For advanced custom trust features, require deliberate configuration from connection settings instead of one-tap bypass from an error alert.

Certificate details can help diagnostics

For an advanced connection inspector, useful non-secret fields may include:

subject / SAN names
issuer
valid-from
valid-until
fingerprint
trust result

Do not imply that displaying a fingerprint makes it trustworthy unless the user verifies it through a separate trusted channel.

Pinning needs rotation support

If you implement pinning, plan before launch for:

current pin
backup pin
rotation window
pin mismatch recovery
clock/time issues

A single hard-coded leaf certificate pin with no backup can turn normal certificate renewal into an outage.

Do not persist arbitrary trust decisions globally

If a user chooses to trust a custom endpoint, avoid a global state such as:

allowUntrustedCertificates = true

Prefer:

connection A -> normal
connection B -> pinned fingerprint X
connection C -> private HTTP explicitly allowed

Trust belongs to the connection.

Test custom trust behavior

Useful test cases:

valid public cert
expired cert
wrong hostname
unknown CA
self-signed cert
pin match
pin mismatch
redirect to same host
redirect to different host
HTTPS -> HTTP downgrade
HTTP local endpoint allowed
HTTP public endpoint rejected

Also verify credential forwarding behavior on redirects and trust failures.

Threat-model local development shortcuts

Developers often use:

mkcert
self-signed certs
reverse proxies
Charles / Proxyman-style debugging
local HTTP

These are legitimate workflows.

The production architecture should still distinguish them from normal trusted provider traffic.

Do not let debugging convenience become a permanent insecure default.

Where BYOKchat fits

A BYOK client that supports both public providers and private custom servers needs more than one networking policy.

A clean design has explicit modes:

public HTTPS with system trust
custom HTTPS with system trust
advanced pinned/private trust when deliberately configured
private-LAN HTTP when explicitly allowed

Credentials, redirects, and diagnostics should respect the selected trust mode without weakening unrelated connections.