How to Design AI Conversation Search

Once an AI client stores more than a few conversations, search becomes a core product feature.

The user is rarely searching for a database record ID. They remember fragments such as:

"the Swift concurrency answer from last week"
"the chat where I compared Claude and Gemini"
"the conversation with the PDF about taxes"
"the answer that mentioned Retry-After"

A useful conversation search system has to bridge that gap without turning private chat history into an unnecessary cloud indexing service.

Start with local lexical search. Add semantic search only when it solves a real retrieval problem and its privacy/cost model is explicit.

Define what is searchable

A conversation contains more than message text.

Possible searchable fields include:

conversation title
user messages
assistant messages
project name
project instructions
attachment filenames
extracted attachment text
tool names
tool result summaries
provider/model labels
tags or favorites

Not every field should have the same weight or even be indexed.

For example, raw tool results may be huge and noisy.

Separate search corpus from display metadata

A search document can look like:

interface SearchDocument {
  id: string
  conversationId: string
  messageId?: string
  kind: "conversation" | "message" | "attachment"
  text: string
  createdAt: string
  role?: "user" | "assistant"
  projectId?: string
}

The index stores retrieval-oriented text and references.

The primary database remains the source of truth for the actual conversation.

Lexical search is a strong baseline

Lexical search handles exact or near-exact terms well:

"URLSession"
"429"
"MCP"
"Gemini"
"Retry-After"

A local full-text index can support:

• tokenization;
• prefix matching;
• phrase search;
• ranking;
• snippets;
• filtering.

It is fast, cheap, deterministic, and private when stored on-device.

Do not underestimate title search

Conversation titles often summarize the user’s memory of a chat.

A useful ranker can give title matches more weight than body matches:

exact title match > title prefix > user-message phrase > assistant-message term

The exact weights should be tested with real usage rather than invented as universal constants.

User messages often deserve more weight

The user is more likely to remember what they asked than the exact wording of a model answer.

For a query:

"local ollama iphone"

a past user message containing those terms may be a stronger hit than one assistant response that mentioned them incidentally.

Keep ranking explainable enough to tune.

Search result granularity matters

Two common designs are:

Conversation-level results

Chat: Connect iPhone to Ollama
Matched 3 messages

Message-level results

Chat: Connect iPhone to Ollama
"...use the Mac's LAN address instead of localhost..."

Message-level results are usually more useful once conversations become long.

The UI can still group multiple hits under one chat.

Snippets should come from the matched region

Do not show only the first 150 characters of the message.

If the query matches near the end, construct a snippet around the hit:

...the iPhone cannot use the Mac's localhost address because loopback refers to the phone itself...

Highlight matched terms carefully without corrupting Markdown rendering.

Index normalized text, render original text

Search can normalize:

case
Unicode forms
punctuation for tokenization

But the result should display the original message content.

Do not replace stored conversation text with search-normalized text.

Markdown needs text extraction

Assistant messages may contain:

### Heading

`code`

[link](https://example.com)

For search, extract a plain-text representation.

A useful index may include:

Heading code link

while avoiding raw Markdown syntax noise.

Code-heavy users may want code tokens preserved, so do not strip code blindly.

Code search needs different tokenization instincts

Developer conversations contain identifiers such as:

URLSessionConfiguration
Mcp-Session-Id
response.completed
foo_bar
camelCaseName

A tokenizer optimized only for natural language can make technical search frustrating.

Consider preserving:

• punctuation-bearing protocol tokens;
• camelCase segments;
• snake_case terms;
• file paths;
• error codes;
• model IDs.

Filters reduce ranking pressure

Useful filters include:

project
provider
model
date range
has attachments
has tools
favorite/pinned
role

A user looking for a chat from one project should not need the relevance model to infer that context from text.

Date filtering should use conversation semantics

Decide whether:

createdAt
lastMessageAt
matchedMessageAt

controls date filtering.

For message-level search, matchedMessageAt is often the most precise.

For conversation browsing, lastMessageAt may feel more natural.

Expose one consistent rule in the UI.

Branch-aware search needs an explicit policy

A branch-capable conversation can contain hidden sibling responses.

Two policies are valid.

Search selected paths only

Benefits:

• less noise;
• matches what users normally see;
• avoids resurfacing rejected answers.

Search full branch history

Benefits:

• alternate answers remain recoverable;
• edited prompts are not lost;
• model comparison branches are searchable.

If searching hidden branches, label them:

Alternate branch

and navigate to that branch deliberately.

See How AI Chat Branching and Regeneration Should Work.

Deleted branches must disappear from the index

Search indexes are derived data.

When a branch or conversation is deleted:

primary data delete -> index delete

should be part of the same lifecycle.

A user should not delete a private message and continue seeing snippets from a stale search index.

Index updates should be incremental

Rebuilding every conversation after each streamed token is wasteful.

A practical lifecycle:

streaming draft -> not indexed or temporary
assistant completes -> index final message
message edited -> replace indexed document
branch deleted -> remove documents
conversation deleted -> remove all documents

For long streams, indexing only after semantic completion is simpler.

App crashes require index reconciliation

If the app crashes after saving a message but before indexing it, search can become stale.

Use one of these patterns:

transactional index update
operation journal
last-indexed revision
rebuild missing documents on startup

The primary conversation store remains authoritative.

Schema/version the index

Tokenizer or ranking changes may require rebuilding.

Store an index version such as:

searchIndexVersion = 4

On upgrade:

if storedVersion < requiredVersion:
    rebuild from primary data

Do not write complicated migrations for derived indexes when rebuilding is safe and bounded.

Semantic search solves different queries

Lexical search may miss:

query: "phone cannot reach mac server"
message: "localhost points back to the iPhone itself"

Semantic embeddings can connect related meaning without shared vocabulary.

But this introduces new questions:

• where embeddings are computed;
• whether conversation text leaves the device;
• which model/version generated vectors;
• index size;
• deletion;
• rebuild cost;
• ranking fusion.

Semantic search should not silently upload private chats

If embeddings are generated by a remote API, conversation text is sent to that provider.

That is a material privacy boundary.

A local-first client should either:

use a local embedding model

or clearly obtain user consent for remote semantic indexing.

Do not market local chat storage while quietly cloud-indexing every message.

Hybrid search is often stronger than semantic-only

Technical queries benefit heavily from exact terms.

A hybrid pipeline can combine:

lexical score
semantic similarity
metadata filters
recency signal

For example:

final = fuse(BM25_results, vector_results)

Rank fusion avoids assuming scores from different retrieval systems are directly comparable.

Do not overfit ranking to recency

Recent chats are often useful, but a query for:

"Mcp-Session-Id"

should still return an older exact technical match above a recent unrelated conversation.

Use recency as a modest signal or tie-breaker, not a substitute for relevance.

Search can benefit from conversation titles as separate documents

One approach:

conversation-title document
message documents
attachment documents

Then fuse them into grouped results.

This allows a short title hit to compete correctly with long body messages.

Attachment indexing needs provenance

If the app extracts text from a PDF, index entries should retain:

conversationId
attachmentId
page/section when available
extraction version

A search result can then navigate to the relevant attachment or conversation.

Never merge extracted file text into the assistant message as if the model authored it.

See How File Attachments Flow Through AI APIs.

File extraction can leak secrets into derived state

A user may attach sensitive documents.

If you build:

• OCR text;
• embeddings;
• thumbnail caches;
• extracted metadata;

those are still user data.

Deletion must remove them too.

See How to Delete AI App Data Correctly.

Tool results can overwhelm search

A tool might return:

50,000 lines of logs

Indexing the raw result can dominate the corpus.

Options:

index tool name + bounded textual summary
index only user-visible tool output
exclude raw binary/large payloads

The policy should be explicit.

Search results should not execute anything

A snippet may contain:

[Click here](javascript:...)

or prompt-injection-like text from a tool result.

Search rendering should treat result text as inert content.

Never execute tool calls or arbitrary URLs because a search hit was selected.

Query parsing should stay predictable

Power features can include:

"exact phrase"
project:work
provider:anthropic
before:2026-08-01
has:attachment

But a simple search box should still work without syntax knowledge.

Do not turn ordinary natural-language queries into a fragile mini-language unless users need it.

Search history can itself be sensitive

If you store recent queries, they may reveal:

medical topic
company name
secret project
personal event

Treat query history as user data.

Offer clear/delete behavior or avoid persisting it.

Search analytics should not collect raw queries by default

A privacy-preserving telemetry event might record:

{
  "event": "conversation_search",
  "result_count_bucket": "1-5",
  "latency_ms_bucket": "<100",
  "mode": "lexical"
}

without collecting:

query text
matched message
conversation title
attachment filename

See Privacy-Preserving Analytics for AI Apps.

Search quality needs a test set

Create representative queries such as:

exact API error code
partial title
old conversation
camelCase symbol
phrase in assistant answer
phrase in user prompt
hidden branch
attachment text
Unicode/non-English text
misspelling
semantic paraphrase

Define expected top results.

This makes ranking changes measurable.

Test multilingual behavior

Tokenization differs across languages.

A search system built only around whitespace-separated English tokens may perform poorly for other scripts.

Use platform/database tokenizers with multilingual support when possible, and test with the languages your app expects.

Search should stay responsive while indexing

Large imports or backup restores can create thousands of new records.

Avoid blocking the main UI while rebuilding the index.

A useful pattern:

restore primary data
mark index rebuilding
serve partial/previous search if safe
rebuild in background/bounded batches
atomically swap ready index

Backup restore should rebuild derived search state

A backup does not need to carry a trusted search index.

Restore primary messages and attachments, then rebuild.

This avoids importing stale or version-incompatible index files.

See Backup and Restore Security for Local AI Chats.

Search and sync need stable IDs

If future versions synchronize conversations across devices, index records should refer to stable primary IDs rather than local row offsets.

Then synchronization can update:

message ID M -> revised content

and the index can replace the corresponding document deterministically.

A practical local-first architecture

The index retrieves IDs; the primary database supplies authoritative content.

Useful invariants

search never changes conversation state
search indexes contain no credentials
all result IDs resolve to current primary data
conversation deletion removes derived search documents
index rebuild can be performed from primary data
hidden branch policy is explicit
remote embedding is never enabled silently
raw search queries are not required for telemetry

Where BYOKchat fits

A local-first chat client can make conversation search especially useful because the user may accumulate chats across many providers and models while keeping the corpus on-device.

Lexical indexing can cover titles, messages, projects, and local attachment text without sending the search corpus to a central service. Semantic search can be added as an explicit optional layer rather than a prerequisite for basic retrieval.

Further reading

• Designing a Local-First AI Chat Architecture
• How AI Chat Branching and Regeneration Should Work
• How File Attachments Flow Through AI APIs
• Privacy-Preserving Analytics for AI Apps
• How to Delete AI App Data Correctly