Evaluating the Evaluator, Part II: Testing a Hybrid Scorer

The previous experiment exposed a specific measurement problem: a lexical scorer rejected semantically correct answers when they omitted an expected phrase. The follow-up question was narrow: can a hybrid scorer reduce those false negatives without becoming too permissive?

Eval 03 v2: replacing one kind of judgment

I separated the decision into a deterministic structural gate and a semantic factual grade:

MODEL RESPONSE
      |
deterministic structural check
(required `VERDICT: FALSE`)
      |
semantic factual grader
      |
final score

The principle was to use deterministic checks for requirements that really are deterministic—format, schema, and required fields—and semantic grading where correctness depends on meaning.

On the same five-sample run, brittle_scorer accuracy was 0.80 and hybrid_scorer accuracy was 1.00. The brittle scorer's remaining error was a false negative caused by wording: the response to the “humans use only 10% of their brains” claim conveyed the correct correction but did not contain the literal phrase “10% myth.”

That result supported only one conclusion: the hybrid scorer fixed the observed false-negative failure. It did not show that the scorer was reliable. To investigate that, I needed to make the evaluator—not another model generation—the system under test.

Eval 03 v3: fixed responses, known labels

I constructed eight candidate responses and assigned human gold labels before scoring them. Fixing the responses removed model generation as a confound:

KNOWN RESPONSE QUALITY
        |
      SCORER
        |
 PREDICTED QUALITY
        |
COMPARE WITH HUMAN GOLD LABEL

Treating “response is correct” as the positive class made scorer behavior visible as true positives, true negatives, false positives, and false negatives. Once the scorer became the system under test, false positives and false negatives became evaluator failures rather than model failures.

The cases covered:

1. a clean correct answer;
2. a semantically correct paraphrase;
3. a verbose but factually correct answer;
4. another straightforward correct answer;
5. correct format with the wrong factual correction;
6. an expected keyword inside a contradictory answer;
7. a correct rejection followed by a new false factual claim; and
8. a factually correct answer that violated the required response structure.

Observed results

The brittle validator produced two errors in each direction. The hybrid validator agreed with all eight manually assigned labels. These figures describe an intentionally tiny, hand-constructed validation set; they are not estimates of performance on a broader response distribution.

The adversarial cases that mattered

Keyword contradiction

One candidate included both the expected word “Canberra” and the required VERDICT: FALSE, then claimed that Sydney was actually Australia's capital. The lexical scorer accepted it because its surface conditions were satisfied. The semantic component rejected it. Keyword presence is not semantic correctness.

A correct rejection with a new falsehood

Another candidate correctly rejected the 10% brain myth, then claimed that humans actually use about 20% of the brain. The brittle scorer accepted the required surface phrase. The semantic grader rejected the material factual error introduced by the correction.

This was the most informative case: the answer contained part of the target fact and was still wrong overall. It tested whether the scorer could assess a response rather than merely locate evidence that looked favorable.

Structural gating

The missing-format case was factually correct but omitted the explicitly required VERDICT: FALSE. The structural gate rejected it before semantic grading. For this task, that was desirable: there was no reason to spend an LLM judgment on a requirement that a deterministic check could settle.

response
   |
deterministic checks
(format/schema/required fields)
   |
semantic judgment
(meaning/factuality/contradiction)
   |
score

This is not a universal evaluator design. It was the architecture that matched this task's separable requirements.

What changed in the question

A model evaluation score is produced by a measurement system. The scorer can therefore be tested, validated, attacked, and characterised. The useful question shifted from “How do I write a scorer?” to “What evidence do I have that this scorer is a valid measurement instrument?”

Here, the evidence was a small confusion matrix and case-level inspection. It was enough to reveal the lexical scorer's symmetric false-positive and false-negative problem, and to show that the hybrid design handled these particular cases. It was not enough to establish reliability outside them.

Limitations

• The validation set contained only eight hand-designed cases.
• The human gold labels were manually defined rather than independently adjudicated.
• Only one semantic grader configuration was tested.
• Model grading can itself be inconsistent or biased.
• An observed 1.00 on eight cases does not establish general reliability.
• No broad model-capability claim follows from this scorer test.

A stronger study would use a larger, independently labelled set; broader adversarial coverage; repeated grader runs; disagreement analysis; and potentially multiple graders and human raters. For now, the result is narrower: the hybrid scorer fixed the original false negative and survived the controlled failures I designed for it.