Measuring an AI assistant instead of guessing

The answers looked plausible

The assistant on this site answers questions about my work from a knowledge base of projects, skills, technologies, and certifications. It reaches that evidence through a Model Context Protocol (MCP) server. After a few weeks it had answered twenty-one questions from real visitors.

The failures followed three patterns. Questions about work were answered with learning credentials instead of projects. Course dates were treated as evidence of professional experience. Requests to contact me ended in a refusal without a useful next step.

The obvious response was to strengthen the system prompt. But these failures could have started in the knowledge base, retrieval, or answer policy. Before changing any of them, I needed a way to identify the responsible layer and detect regressions elsewhere.

A repeatable test

I turned those failure patterns into twelve questions. Four came directly from real traffic. Four covered what I want visitors to understand about my work. The final four were safeguards: questions that a narrow fix might accidentally make worse, including one where a certification is the right answer.

Each answer was checked in two ways. Ten simple rules caught clear failures, such as naming a certification when the question asked about projects or expanding an acronym that the evidence did not define. A separate model reviewed qualities that needed judgement. It did not know which version of the system had produced the answer.

I then changed one part of the system at a time and left the model upgrade until last. Changing the model first might improve everything at once, but it would hide the value of the work on content, retrieval, and answer policy.

The first retrieval change did nothing

The assistant was finding relevant projects but ranking them too low. On the React question, the first project appeared at rank 18.

I first changed two settings that appeared to control this behaviour. The next run was identical. Both settings belonged to the resume builder, which shares retrieval code with the assistant but follows a different path. The assistant never used them.

The useful change had to happen later, after the search results were combined. My first attempt gave projects too much weight. The original scores were close enough that a 35% boost could move the least relevant project above the most relevant certification. A safeguard question caught that before it shipped, and I reduced the weight until relevant projects moved up without breaking the certification case.

A stable number exposed a bad metric

I expected a stronger answer policy to reduce irrelevant certification references. The reported count did not change across five versions of the system, so I inspected the measurement instead of rewriting the policy again.

I had been counting every certification made available to the assistant, not the ones it used in its answer. The policy could affect the answer, but it could never change the evidence returned by retrieval. Once I separated those two stages, the results showed that the model had been filtering certifications all along.

Because each run stored the original answers, I could correct the metric and regrade the earlier results without running the assistant again.

One result was not enough

I tried giving professional projects priority over personal ones so that general questions would lead with paid work. The first run looked much worse than the previous version.

Repeating the unchanged version showed that the score itself varied between runs. When I compared the individual samples instead of the overall score, the difference fell to about three points—too small to separate from the normal variation.

I still removed the change because it placed employment status ahead of relevance. On AI questions, professional projects with no AI content displaced personal projects that answered the question directly. The measurement did not prove a regression, but the design was still the wrong trade-off.

What improved

The first comparison tracked the system as I improved its content, retrieval, and answer policy. Each result used three samples per question:

I then compared both models with everything else held constant, using ten samples per question:

Luna won or tied every question and was more reliable about keeping certifications out of unrelated answers. The trade-off was a slower response.

The two tables are separate because they used different sample counts. More samples make the test stricter by creating more opportunities for an answer to fail.

The most effective fix was in the data

The knowledge base links technologies to the projects that use them. Some of those links were missing: React was connected to only four of the eight relevant projects.

Ordinary text search still found the projects, which hid the problem. The lookup based on relationships could not reach them.

Repairing the links moved the first project for the React question from rank 18 to the top of the results. The best retrieval improvement was not a new algorithm. It was a correction to the source data.

What I learned

Safeguard questions are worth keeping. They caught two changes that improved the target questions while making another case worse.

Search results and final answers also need separate measures. One shows what evidence was available; the other shows what the model chose to use. Keeping the original answers means a faulty metric can be corrected without repeating the whole test.

Several diagnoses that felt obvious turned out to be wrong. I only know because I stopped judging plausible answers by eye and built a repeatable test.