Why Kai searches before it answers

Kai is the chat on this site. It answers questions about my work and cites where each answer comes from. The first version of Kai was a chatbot with my resume pasted into the prompt. This one searches my content first.

Why search comes first

A chatbot can have a search tool and still answer without using it. Kai runs on low-cost services, and models vary in how reliably they look up facts. I don’t want the quality of an answer to depend on whether the chatbot chose to search.

What runs instead

Before Kai writes anything, the server searches my pages. It uses the visitor’s latest message and the one before it, so a follow-up like “and what about the second one?” still has context. It supplies the five best passages as numbered sources, S1 to S5.

Kai writes an answer with those numbers. The server checks each one against the passages it supplied. It drops any number that does not match. An answer left without a valid source becomes “I don’t know that one.”

Most answers need one request to the chat service. Kai can ask for up to two extra searches, but the first search never depends on it doing so.

How the search works

I build the search index from this site’s pages before deployment. Each passage is a section of a page, with a header like “Case study: Over-the-air updates › Result”, so it makes sense alone. Two searches rank the passages, one by keywords and one by meaning. I merge the two ranked lists.

When a question has at least three distinct terms and the best keyword match contains all of them, the server skips meaning-based search. That saves a network request. If meaning-based search fails or takes more than two seconds, keyword search works alone. I avoid mixing meaning-based scores from different search services, because those scores cannot be compared reliably.

How to check a source

You don't have to take the citations on faith. Each citation in an answer is a numbered chip. Open it and the passage it points to appears under the answer, with a link to the full page.

If an answer is wrong, the passages show why. Either the right passage never came back, or it came back and the model misread it. Those are different bugs, and the passages tell me which one I have.

What I haven't measured yet

I haven't published test results. The plan is a set of 40 questions that checks required facts, forbidden facts and citations, including trap questions about things I don't share. The scores will go on a public page when they exist.