Evaluating a system on your own cases

Public rankings compare models on general tests. They say little about how an assistant will handle your contracts, your procedures or your customers’ questions. The researchers behind the HELM project argued for evaluation across many scenarios and several criteria at once: accuracy, robustness, fairness, cost.

1. Gather about twenty real, varied requests, including difficult cases.
2. For each one, write the expected answer or the points it must contain.
3. Add two or three cases whose answer is not in the sources, to check that the assistant says so.
4. Run the cases and grade each answer: correct, partial, wrong, or justified refusal.

This set of cases becomes your control tool. When you change an assistant’s instructions, add a source or move to another model, run the same cases again and compare. An improvement on one point must not break another.