How Do You Grade an Answer with No Correct Wording?

The SE coaching bot has four skills. One of them (qualify) produces structured JSON with eight scored elements, so grading it is arithmetic: compare the model's scores against a ground-truth front matter and compute agreement. The other three (objection handling, competitive positioning, demo planning) answer in prose. There is no correct wording. An answer can land the right substance in completely different words, or it can sound fluent and authoritative while asserting things the corpus never said.

The problem with grading coaching answers

A coaching answer has no arithmetic ground truth. It is not a classification task with a label, or a structured output with a schema to validate against. Two good answers to the same objection can take entirely different approaches, emphasise different evidence, and still both be correct. So "did the model get the right answer" is the wrong question. The questions that can be answered are: did the answer contain the substance the corpus supports, did it commit a known failure, and did it invent anything the context does not support.

Those three become the areas of verification and Point Coverage measures whether the expected substance appeared. Trap Rate measures whether named failures were committed. Invention Rate measures whether the answer's factual claims are grounded in the context the engine actually supplied. Each axis judges a different thing, and collapsing them into a single score would mask the diagnostic that makes the eval useful.

Writing the marking scheme

Each scenario carries a front matter with expected points, named traps (each with a why explaining the error and why it is tempting), grounding sources, and whether a refusal is expected. The body is rationale for the next human reader; the eval never reads it.

The points are authored from the corpus, not from a model's answer. That distinction matters because if I wrote the points by running the model and then describing what it said, the eval would measure whether the model is consistent with itself, not whether it surfaced what the corpus supports. Every point has to be traceable to a passage in content/kb/.

The traps are where the rubric earns its keep. Point coverage alone is gameable by thoroughness: an answer that recites every relevant fact in the corpus scores well on coverage while being exactly the answer the corpus's meta-coaching section warns against. The trap axis catches that. Each trap names a specific wrong move (not a vague "don't be wrong") and carries a why field that goes into the judge prompt, because the tempting version of a wrong answer usually looks defensible in isolation, and a judge that does not know why it is wrong will mark it generously.

Two scenarios per skill are out-of-corpus: the request sits outside what the Knowledge Base can answer. These test whether the model declines honestly rather than improvising. Refusal correctness is scored over these two alone, which makes it the coarsest metric in the set (it can only read 0.00, 0.50, or 1.00), but each individual failure deserves reading.

Two judges, two jobs

Each scenario gets two judge calls, separated because they need different evidence.

The coverage judge sees the request, the answer, the expected points, and the traps. It scores each point PRESENT / PARTIAL / MISSING and each trap AVOIDED / PARTIAL / HIT. Verdicts are returned as brief locators ("the deployment-choice sentence", "the bulleted next steps") rather than quotations, because one unescaped quote character inside a JSON string makes the entire verdict set unparseable, and that costs every verdict, not just the one quote. This was discovered the hard way: the grounding judge was initially truncating at 8192 tokens, but the real cause turned out to be unescaped quotes in evidence fields corrupting the JSON parse.

The grounding judge sees the request, the answer, and the retrieved context the engine actually used. It extracts substantive factual claims and classifies each one as GROUNDED, PARTIALLY_GROUNDED, UNGROUNDED, or NOT_A_CLAIM. The request is included so the judge can distinguish an echo of the SE's own premise from an assertion the assistant is making; without it, the invention rate was inflated by roughly a third because every restatement of the user's question scored as ungrounded.

The judges are instrumentation, not graded models, so their output parsing is deliberately lenient: a verdict wrapped in a code fence or surrounded by prose is recovered rather than discarded. The graded model's own output is never treated this leniently, because its conformance is the measurement.

Inter-judge agreement

A second judge runs the coverage call independently when --secondary-judge is set. Agreement is the share of verdicts they scored identically. It is a property of the rubric rather than of the model: if two independent judges stop agreeing, every other number on the scorecard stops meaning anything. That is why it is gated; a run with agreement below 0.70 fails the threshold.

The objection set took nine calibration runs to settle. These runs surfaced several bugs beyond the rubric itself: the grounding-judge evidence truncation, the cache replaying stale results after judge configuration changed, the grounding_ok rehydration default assuming absence of an error flag meant success, and the refusal metric going through three iterations before the invention-rate clause was correctly balanced (removed, then restored after a positioning scenario invented regulatory alignment while avoiding all traps). Positioning and demo planning were authored afterwards against four rules learned from the objection calibration: no trap that negates a point (they cancel each other's signal), the declining-honestly point as p1 (because a model that refuses cleanly should score something before it is penalised for not answering), traps naming the specific form of a failure rather than vague "gives bad advice", and no expected point the corpus cannot support. Both produced usable agreement on the first run (0.90 and 0.86).

Cache integrity

A graded run over one skill is twelve engine calls plus twenty-four to thirty-six judge calls, and it takes ten to fifteen minutes. Without checkpointing, a throttle at minute fourteen loses all of it. Every result is appended to a JSONL cache as it lands, and a re-run loads what exists and skips those calls.

The cache key covers the graded model, the judges, a hash of both judge prompts and their token budget, and a hash of the scenario set's marking scheme. Any of those changing misses the cache on its own, without needing --fresh. That design exists because the opposite happened: a max-tokens change was cached straight past during calibration, and the resulting run reported a clean grounding-failure rate for a run in which a third of the grounding calls had silently failed through truncation. The fingerprint is derived rather than hand-maintained, because a version constant that must be bumped by hand is a constant that will not be.

What the key deliberately does not cover is the scoring arithmetic. A scoring change re-aggregates stored verdicts for free, which matters because the arithmetic moved several times during calibration. That split is the whole design: pay again for a different question, never for the same question scored differently.

Results

All three skills under one configuration: Sonnet 4.6 as the graded model, GPT-oss-120b as the coverage judge, Haiku 4.5 as the agreement judge, twelve scenarios per skill.

Coverage is high and consistent across skills. Trap rate and invention rate are low, with demo planning the weakest on both axes but still within threshold. The thresholds were calibrated on the objection set alone and transfer to the other two skills unchanged, which is some evidence they describe the system rather than one scenario set.

Refusal correctness breaches on objection and positioning. The breaches are real and deliberate: lowering the bar to make the run green would delete the most useful line on the scorecard.

The failure I didn't expect

The dominant failure pattern across all three skills is one I had not anticipated: refusing the artefact and then improvising the substance.

The model reliably declares the gap in out-of-corpus scenarios and then supplies content anyway. In the objection set, it declines the article-by-article regulatory mapping and then claims its controls supply "relevant ingredients for an AI Act compliance story", which is the same claim at theme level rather than article level. In positioning, it declines to give pricing figures and then characterises a competitor's commercial model as though the corpus had established it. In demo planning, it says plainly there is no content on demo environments and then invents environment types, a lead-time claim, and advice on how late is too late to change. Four of five claims ungrounded, two traps hit, and the worst single scenario in the set.

This is subtler than a flat hallucination and harder to catch by inspection. Every individual sentence reads as careful and helpful, and the disclaimer at the top makes the answer look more trustworthy rather than less. It is the pattern a manual spot-check is least likely to catch, because the manual reader sees the disclaimer and stops scrutinising.

Caveats

Judged levels are judge-dependent. The absolute values are a property of the judge as much as of the answers. Compare models under the same judge and trust deltas, not levels.

Invention rate is not comparable across judge configurations at all, not even as a delta. The judge decides how many claims to extract, so it chooses its own denominator. The same answers scored 103 claims under one judge and 58 under another.

Roughly two in nine claims flagged as invention are artefacts: a restatement of the request, the refusal's own statement about what the corpus lacks, or the answer quoting what not to say. The grounding judge is given a NOT_A_CLAIM label to reduce this. The label has been used zero times across three runs and thirty-six scenarios, so it is a backstop that does not fire. Treat the artefact rate as a floor on the measurement's noise.

The engine runs at default temperature, so answers differ between runs. Coverage moving a point or two between runs is noise, not signal.

What this changed about how I think about evals

The eval framework and the complaints bot both graded outputs where a correct answer exists: a factual claim is either supported or it is not, a provision is either cited correctly or it is not. The coaching eval does not have that luxury. An answer with no correct wording can only be graded against a scheme that says what substance should appear, what failures should not, and what facts require grounding; and the scheme itself has to be calibrated until two independent judges agree on what it means. The rubric is the deliverable as much as the code is.

The refusal-improvisation finding was the most useful output of the whole exercise, and it came from the traps, not the points. Point coverage would have scored those scenarios respectably. Without a trap that says "asserts a specific certification the corpus does not contain" and a grounding judge that checks every factual claim against the retrieved context, the pattern is invisible. It is the combination of the two axes that makes it visible, and collapsing them into a single quality score would have hidden it.

The scorecards and the scenario sets are in the repository.