Privacy-Preserving Analytics for AI Apps

AI apps benefit from analytics.

You may need to know:

which providers fail most often;
how long requests take;
whether streaming stalls;
how often tools execute;
whether onboarding completes;
whether a model capability check is wrong;
whether a new release increased crash rates.

None of those questions require collecting the user’s conversation.

The strongest privacy design is:

Define exactly which operational facts are useful, then make sensitive AI content impossible to serialize into analytics events.

Start with purpose, not available data

A common analytics failure begins with:

What can we collect?

A better question is:

What product/reliability decision will this event support?

If you cannot name the decision, do not collect the field.

Separate telemetry from conversation storage

Your app may store conversation text locally because that is the product.

That does not mean analytics needs a copy.

Model these as separate systems:

The analytics pipeline should not have access to raw chat content unless there is an explicit, justified feature requiring it.

Useful metrics rarely need text

For request observability, fields such as these can be enough:

{
  "event": "ai_request_finished",
  "provider_kind": "openrouter",
  "result": "success",
  "status_class": "2xx",
  "input_tokens": 1420,
  "output_tokens": 380,
  "ttft_ms": 720,
  "duration_ms": 5810,
  "tool_calls": 1,
  "stream_interrupted": false
}

This helps you analyze performance and reliability without storing:

prompt
response
reasoning
tool arguments
tool results

Prefer categorical provider data over account identity

Useful:

provider_kind=anthropic
connection_type=custom_https
model_capability=reasoning

Usually unnecessary:

user email
provider account ID
full custom hostname
API key fingerprint

Do not turn operational telemetry into an account directory.

Custom endpoints need extra care

A hostname can reveal sensitive information.

Examples:

llm.internal.company.example
patient-ai.hospital.example
192.168.1.42

If analytics only needs to know transport class, record:

custom_endpoint=true
transport=https
private_network=true

instead of the full URL or hostname.

Never collect credentials

Reject fields containing:

API keys
Authorization headers
OAuth access tokens
refresh tokens
protected custom headers
passwords
private keys

This should be a schema invariant, not a documentation suggestion.

Avoid prompt-derived identifiers

A tempting technique is:

hash(prompt)

so you can “deduplicate without storing text.”

This still creates a stable identifier derived from private content.

For short/common prompts, correlation or guessing may be possible.

If the product does not need prompt identity, do not create one.

Generate local opaque IDs instead

If you need to correlate events across one request or conversation, generate random identifiers:

request_id = UUID()
conversation_analytics_id = random opaque value

Do not derive them from:

prompt
API key
hostname
email
file name

Conversation-level analytics can stay content-free

Example:

{
  "event": "conversation_updated",
  "turn_count_bucket": "11-20",
  "provider_switches": 1,
  "attachments_used": true,
  "tools_enabled": true
}

You can understand feature usage without learning what the user discussed.

Bucket values when precision is unnecessary

Instead of storing exact values forever:

turn_count=137
context_tokens=82341

consider buckets:

turn_count=101+
context_bucket=64k-128k

Bucketing can reduce fingerprinting while preserving the product signal you actually need.

Timing data is usually low sensitivity, but context matters

Metrics such as:

TTFT
duration
tokens/sec
network failure count

are valuable and usually do not reveal content directly.

Still avoid combining many stable dimensions into a highly identifying profile when aggregate analysis is enough.

Distinguish local analytics from transmitted analytics

An app can maintain detailed local analytics without sending them anywhere.

For example:

per-provider token totals
estimated spend
TTFT history
request success rate
tool-call count

The user can inspect this on-device.

Only a smaller sanitized subset needs to leave the device for product telemetry, if any.

Keep analytics schemas typed and narrow

Bad API:

analytics.track("event", properties: [String: Any])

Any caller can attach arbitrary sensitive state.

Safer:

struct RequestFinishedEvent {
    let providerKind: ProviderKind
    let result: ResultClass
    let ttftMs: Int?
    let durationMs: Int
    let inputTokens: Int?
    let outputTokens: Int?
}

Typed event models make privacy review much easier.

Unknown fields should fail closed

If an analytics serializer receives a new property that is not in the approved schema, reject it rather than forwarding it automatically.

That prevents a future refactor from accidentally adding:

requestBody
rawURL
authHeaders

to production telemetry.

Sanitize errors into categories

Raw error strings may contain:

endpoint URLs;
request IDs;
file paths;
provider payloads;
user input.

Map them to safe categories:

network.timeout
network.offline
tls.failure
http.401
http.403
http.429
provider.invalid_response
stream.interrupted
tool.validation_failed

Then retain a local detailed error if the user needs diagnostics.

Tool analytics should record behavior, not arguments

Useful:

{
  "event": "tool_call_finished",
  "tool_kind": "calendar.create_event",
  "approval": "asked",
  "result": "success",
  "duration_ms": 54
}

Avoid:

event title
recipient
email body
file path
search query
URL

Tool arguments are often more sensitive than chat text.

Attachments need metadata minimization

You may want to know whether file workflows are used.

Safe-ish aggregate fields can be:

attachment_count
attachment_size_bucket
attachment_kind=pdf/image/text

Usually avoid:

filename
path
file contents
extracted text

unless the feature specifically requires them.

Reasoning should stay out of telemetry

Reasoning state can contain private intermediate information and provider-specific artifacts.

You can measure:

reasoning_enabled
reasoning_effort=high
reasoning_tokens=provider_reported_count

without collecting reasoning text.

Sampling can reduce volume and privacy exposure

Not every request needs to be transmitted.

Sampling strategies include:

1% of successful latency events
100% of normalized crash events
aggregate counters for common flows

Choose sampling based on the decision you need to make.

Less data is easier to protect.

Retention should be finite

Even sanitized telemetry accumulates risk.

Define:

how long raw events live
when they are aggregated
when identifiers rotate
when old data is deleted

Do not keep detailed event streams forever merely because storage is cheap.

User controls should match the data model

If analytics is optional, an off switch should stop transmission cleanly.

It should not leave a hidden secondary telemetry path through:

crash breadcrumbs;
debug uploads;
third-party SDKs;
support bundles.

Document which diagnostics are local and which leave the device.

Third-party SDKs expand the trust boundary

A generic analytics SDK may automatically collect:

device identifiers;
screen names;
URL information;
session metadata;
crashes;
network breadcrumbs.

Before integrating one, verify what it collects by default.

A small first-party event endpoint can sometimes be easier to audit than a broad SDK, but you still need retention, security, and operational discipline.

Privacy-preserving analytics can still answer business questions

Examples:

How many users configured >1 provider?
How often does local AI connection validation fail?
Which app version increased 429 errors?
How many requests use reasoning?
How often are MCP tools approved vs denied?
What is median TTFT by provider kind?

None require reading prompts.

Keep cost analytics local when practical

A BYOK client can calculate estimated spend locally from:

provider/model pricing metadata
input/output token counts
cached token accounting

The user benefits directly from that data.

Central product analytics may only need a coarse feature event such as:

cost_dashboard_opened=true

rather than the user’s actual spend.

Test analytics privacy as code

Create fake sensitive fixtures:

TEST_PROMPT_PRIVATE_SENTINEL
TEST_API_KEY_SENTINEL
TEST_HOST_SECRET.internal
TEST_TOOL_ARG_PRIVATE

Run requests, errors, tool calls, backups, and analytics serialization.

Assert the outbound analytics payload contains none of them.

Review analytics changes like API changes

When adding a field, ask:

What decision uses this?
Can it identify the user/account?
Can it reveal conversation content?
Can it be derived locally instead?
How long do we need it?

If the answer is vague, leave the field out.

A practical event boundary

Raw app state should never be serialized directly into the analytics queue.

Where BYOKchat fits

A local-first AI client can provide detailed on-device analytics—tokens, estimated spend, TTFT, duration, generation speed, reliability, and tool counts—while keeping prompts, responses, reasoning, credentials, tool arguments/results, attachments, URLs, and hostnames out of transmitted telemetry.

That separation preserves both useful observability and the reason users choose a BYOK/local-first architecture in the first place.