Evaluating the Evaluator: What a Broken Scorer Taught Me About LLM Evals
Three tiny evaluations moved from exact answers to semantic corrections—and showed why a score can describe the evaluator as much as the model.
Scope: This was a small learning experiment about evaluation methodology, not a GPT-5 benchmark. The sample sizes are too small for broad claims about model capability.
Question
What happens when the task changes faster than the scorer? I wanted to see how three evaluation choices behaved as answers moved from objectively checkable arithmetic to open-ended corrections of false premises.
Why I tested it
An aggregate accuracy can look authoritative even when its scoring rule does not measure the behavior the task asks for. The aim was not to rank a model. It was to learn where deterministic matching, model grading, and a lexical target check agree—and where inspecting the underlying transcripts changes the interpretation.
Experimental setup
I ran three small evaluations:
- Eval 01: five arithmetic samples scored deterministically with
match(). - Eval 02: five open-ended false-premise prompts scored with
model_graded_qa(). - Eval 03: an initial four-sample run scored by lexical target matching, followed by a revised five-sample run.
No public code or logs are linked here because no verified public URL was supplied.
Prediction
I expected exact matching to work for constrained arithmetic, and model grading to be better suited to open-ended corrections. I also expected a simple lexical target rule to be reproducible, but potentially misaligned with semantic correctness.
Result
| Evaluation | Samples | Scorer | Observed result |
|---|---|---|---|
| Eval 01 | 5 | match() | 5/5; accuracy 1.0 |
| Eval 02 | 5 | model_graded_qa() | 5 observed correct grades; accuracy 1.0 |
| Eval 03, initial | 4 | Lexical target matching | Accuracy 0.50; stderr ≈ 0.289 |
| Eval 03, revised | 5 | Revised run | Accuracy 0.60; stderr ≈ 0.245 |
Aggregate metric
The first two accuracy values were both 1.0, but they came from different measurement processes: deterministic matching for five arithmetic answers, and five model-assigned grades for open-ended answers. Equal aggregates did not make the scorers interchangeable.
Eval 03 made uncertainty harder to ignore. Its initial accuracy was 0.50 with standard error approximately 0.289 across four samples. The revised five-sample run reached 0.60 with standard error approximately 0.245. These are descriptive results from tiny runs, not stable estimates of general performance.
Transcript inspection
The initial Eval 03 aggregate hid the most useful finding. Reading the sample-level records showed that two responses made the right semantic corrections but were rejected because they did not satisfy the lexical target match. The scorer marked surface-form disagreement, not substantive error.
This is why transcript inspection was not merely debugging after the metric. It was part of validating what the metric meant.
Failure mode
The broken assumption was that the presence or absence of a target string could stand in for semantic correctness. That rule can create false negatives, as the two rejected corrections showed. It can also create a structural false-positive vulnerability: mentioning the expected term is not the same as correctly resolving the premise.
The Canberra item illustrates that second weakness. Its lexical structure left the scorer vulnerable to accepting the target term without establishing that the response had made the right correction. The observed GPT-5 run did not trigger that vulnerability. It remains a property of the scoring design, not an observed model failure.
Follow-up
The revised run expanded Eval 03 from four to five samples. Its observed accuracy was 0.60 and its standard error was approximately 0.245. That change did not, by itself, validate the scorer; it provided another small result while keeping the scorer-design question visible.
A better next scorer would be hybrid:
- Structural checks verify task-specific requirements that can be tested deterministically.
- Semantic grading assesses whether the response actually identifies and corrects the false premise.
- Recorded component outcomes keep the final decision auditable instead of collapsing every judgment immediately into one bit.
Limitations
- The evaluations contained only four or five samples per run.
- The observations do not establish repeatability across prompts, graders, or runs.
- Eval 02's perfect score is still mediated by a model grader; it is not ground truth merely because it is 1.0.
- The Canberra weakness was identified structurally, but was not triggered by the observed GPT-5 run.
- A hybrid scorer introduces new risks: structural rules can remain brittle, semantic judges can be inconsistent or biased, and conflicts between components require an explicit resolution policy.
Most importantly, this was a small learning experiment about evaluation methodology, not a GPT-5 benchmark.
Next experiments
- Test the hybrid scorer on paraphrases that preserve meaning while varying surface form.
- Add adversarial answers that contain a target term but do not make the required correction.
- Compare structural and semantic component decisions before combining them.
- Repeat the evaluation with larger sample sets and report disagreement and uncertainty alongside accuracy.
- Manually review scorer disagreements to separate model errors from evaluator errors.
The practical lesson is simple: before trusting an evaluation score, evaluate the evaluator.