Our verification layer uses an outside model as a judge: it reads a claim and its source and decides whether to allow it. In July, Mike Czerwinski (@jugeni) asked a question we could not answer, in the comments of his own post on where the line between code and judgment sits. The whole exchange, run by run, is in that thread. Does that judge's verdict change when only the voice of the claim changes? We said we would run it and publish the result either way. This is that post, and the short answer is yes, a little, and for a reason we can now name.
It took seven runs. Two of our own numbers did not survive, and one is still waiting on a re-run. Here is the whole story in one picture, and each section below is one box of it.
1. The question
The judge was chosen so its mistakes would not line up with the mistakes of the model it checks. Mike pointed out that this says nothing about style. A judge can reason independently and still relax when the prose sounds confident and familiar. If the verdict moves when only the voice moves, our independence claim is weaker than we measured.
Three things in this post belong to Mike: the hypothesis, the push to check content with a rule instead of a second model, and the fix we tried at the end. We built the harness, ran it, and are publishing the version where we were wrong.
Our first attempt (v0) used ten cases far from the decision line, and the verdict never moved. Cases that far from the line could never have flipped, so the run could only pass. It also showed us the judge's score takes just two values, so it carries no confidence to measure.
2. The first number was a confound
v1 used twelve cases near the line, each rewritten in three voices. A third of the verdicts flipped when the voice changed, and hedged claims passed more often. We posted that as an early result, and the headline read like "the judge rewards hedging."
It was wrong in the part that mattered. To change the voice we rewrote the text, and rewriting the text changed the content. The confident versions carried more evidence; the hedged ones were vaguer claims as well as softer ones.
The fix was Mike's. Check that content survived with a rule instead of a model, because a model adds a second judge with its own taste in style. So every case became one neutral claim, and each voice is the same claim with "Certainly," or "Perhaps," added. A plain rule checks that every number and content word survives.
3. Holding content fixed
With content locked, the flip rate fell from 33% at n=12 to 17% at n=24 and 14.6% at n=48 (95% CI roughly 4 to 25%). It still sits well above how often the judge disagrees with itself on an identical input.
4. The direction disappeared
The bigger change was in direction. At v1, voice looked like it moved pass rates by 25 points. At v3 the three voices passed at 88%, 88% and 85%. Assertive minus hedged is +2 points with a 95% CI of -8 to +12. The interval straddles zero, so there is no directional tone bias. Content held up too, with cosine similarity to the neutral version at 0.955 for assertive and 0.923 for hedged.
So about 15% of borderline verdicts still flip on tone alone, and they flip in both directions. Please retire the 33%. It was our number, we published it first, and it is the one we most want retired.
5. Is it just noise?
A flip rate means nothing until you know how often the judge flips on an input that did not change. Our first try (v5, 8 repeats) said never. The second (v6, 6 repeats on more cases) found the judge's repeated calls are independent samples, and the true floor is 2.08% on borderline cases and 0% on clear ones. One caveat: between those two runs we swapped the judge model to a larger one, so v5 and v6 measure different judges.
Mike asked for two more controls. Control A ran the three voices on clear cases, not only borderline ones. The effect is smaller there, and still present. "The test suite passed" is allowed when stated plainly and refused when hedged.
Control B asked whether the flips are the same claims every time or random scatter. They are the same claims, and those claims stay put when the tone is held still. Tone walks specific claims across the line.
6. Why it happens
The judge reads how sure the claimant sounds as evidence about whether the claim is supported. That one sentence explains both directions. A hedge makes a weak claim sound careful, so it gets through. A hedge makes a strong claim sound unsupported, so it gets blocked.
For a general reviewer that is reasonable behaviour. For a source-grounding check it is a flaw, because the only question should be whether the source supports the claim. Ours moves with how confident the claimant sounded, and now we can say by how much and in which direction.
7. The fix, and the part we owe you
Mike proposed stripping the stance words before the judge sees the text. On 07-18 we ran it and told him the flip rate dropped to about 2.1%, all the way to the noise floor.
When we went back to write this up, we could not find that run's artifact: no v7 script and no result file anywhere in our history. We are keeping the report, since the session that had just run it made it. We stopped citing it as evidence, because nobody can check it. The line we hold is reported versus re-citable.
The 2.1% also matches the v6 floor of 2.08% exactly. That fits a fix that removed the effect completely. It also fits a number that travelled: a handoff note called 2.1% a floor, and a later summary rewrote it as a fix that worked. We treat that second possibility as a pipeline defect. Running v7 again, with the artifact kept, decides which one it was.
What we still do not know
- One judge. The model swap between v5 and v6 shows how much that can matter.
- A hand-picked fixture. Cases were chosen near the decision line on purpose, so these flip rates describe cases near the line rather than ordinary traffic.
- Added stance words only. Confidence woven into sentence structure is the harder case, and a naturally hedged fixture is the next thing we owe.
- Stripping leaves residue. Our stripper left a dangling "though". It carried nothing here, but leftovers like that are where a residual effect could hide.
The residual flips go both ways. Boundary noise looks like that, and so does a very small bias hiding inside boundary noise. We lack a method that separates the two. If you do, we would like to hear it.
An evaluator is only worth what its error bars are worth. The number we most wanted to be true, 33% and a clean mechanism, is the one we spent three runs killing.
If you want the run-by-run conversation that produced all of this, including Mike's designs for the next fixture, it is in his thread.
Top comments (16)
Separating the missing v7 artifact from the measured runs is the strongest part of this report. For the rerun, I would pin the same judge version and run stripped, unstripped and unchanged-input controls together, rather than comparing a new fix run against a noise floor measured under an earlier configuration.
Because the same claims flip repeatedly, uncertainty should also respect the claim as the sampling unit. Treating every voice or repeat as independent can make the interval look tighter than the fixture warrants. A paired per-claim analysis would show whether stripping reduces disagreement on those particular boundary cases without claiming transfer to ordinary traffic.
Agreed on the rerun, and that is how we will run it. The 2.1% was set against a floor from an earlier run, and a judge swap sat between v5 and v6, so the comparison never shared a configuration. The v7 rerun will be one run with the judge version pinned and recorded, and every claim going through three arms in the same pass: unchanged input repeated, unstripped voices, stripped voices.
On the sampling unit, the published intervals already treat the claim as the unit. The v3 bootstrap resampled whole claims with their three voices together, and the assertive-minus-hedged gap is paired inside each claim. The 2.08% floor is also per claim, 1 of 48. Where your point lands is the stripped-vs-unstripped comparison, which so far has been two rates side by side. The rerun will report it per claim: for each claim, did it flip under tone before stripping, and after, with a paired test on the claims that changed. With about 7 flipping claims out of 48 that test will be thin, so we will say how thin rather than claim transfer.
Thanks for correcting my reading of the published intervals: the claim-level resampling was already there. The new information is the paired stripped-versus-unstripped transition, rather than a change to that earlier sampling unit.
I would show the four transition counts alongside the paired estimate: stable in both conditions, unstable in both, improved after stripping, and worsened after stripping. With only a few changing claims, those counts make the uncertainty tangible and keep a small observed improvement from sounding like evidence of general transfer.
That table is a better report than the one I described, and it costs nothing extra. The four counts are the 2x2 the paired test is built on anyway: it only uses the two discordant cells, improved and worsened, so putting all four on the page shows how few of the 48 claims the estimate rests on. I'll put it in the v7 write-up beside the paired estimate, with the number of changing claims stated in the same sentence as the result.
Putting the changing-claim count beside the paired estimate makes the effective evidence visible without another experiment. I would also keep the per-claim transition labels in the released result, so the four-cell table can be regenerated rather than manually transcribed.
One detail to state in the write-up is the treatment of missing judgments in either half of a pair. Dropping those pairs changes the comparison population; assigning them an explicit unavailable state preserves the distinction between a claim that stayed unchanged and one that could not be compared.
Good catch, and our earlier code got this wrong. In the v6 script a failed judge call was skipped silently: the error was caught, nothing was recorded, and a claim was scored from whatever calls came back. A claim with nothing left just fell out of the averages. It happened to cost nothing there, since every one of the 48 claims came back complete in the saved result, but the script would not have told us otherwise.
The v7 rerun hasn't run yet, so this goes in first. Each half of a pair will record either a judgment or an explicit unavailable state with the error. The table gets a row for pairs that could not be compared, and the estimate says how many claims it excludes. Agreed on releasing the per-claim labels too, so the table can be regenerated instead of transcribed.
Keeping the v6 result and its harness weakness separate is fair: all 48 saved claims were complete, but the code did not establish that as a required condition. The planned unavailable rows will make that condition auditable for v7.
I would retain the two halves' error states separately and report how many complete pairs remain before calculating the paired estimate. That also lets readers see whether missing judgments cluster in one prompt condition, which could bias a complete-pair result even when every excluded claim is counted.
Both points go into v7 before it runs. Each half keeps its own state, so a pair can read judged/judged, judged/unavailable, unavailable/judged or unavailable/unavailable, and the table reports the count in each before any paired estimate is computed. We will also break the unavailable count down by prompt condition. If the misses pile up in one dressing, the complete-pair result is biased toward the easy cases, and that table is where a reader would see it. Thank you for pushing on the order of operations, which is the part I had left implicit.
Those four states give v7 an accounting check before any effect estimate: their counts should sum to the planned pair count, with no pair disappearing during parsing or joining. Keeping the prompt-condition breakdown then shows whether exclusion is asymmetric.
I would preserve the pair identifiers in that table and report the complete-pair estimate with its actual count. If missingness differs by dressing, the appropriate conclusion is conditional on evaluable pairs, rather than treating the surviving sample as if it still represented every planned comparison.
Both go into v7. The table will carry the pair IDs, so a reader can check that the four states add up to the planned pair count and see exactly which pairs dropped out. The paired estimate will be reported as N evaluable pairs out of M planned, and if the unavailable rows lean toward one dressing, the write-up will say the result covers the evaluable pairs only. We relearned the same lesson this week on a different test: a rate printed without its denominator, and without the items that fell out, read as stronger than it was. v7 has not run yet. When it does, that table ships with the result.
The most important finding here is that evaluator independence isn't enough if the evaluator can still use presentation style as an implicit feature. A judge can be completely different from the model it checks and still introduce its own shortcut.
This is something we pay close attention to at IT Path Solutions when designing production AI verification workflows. I’d separate three signals explicitly: content support, evaluator uncertainty, and presentation style. For a source-grounding check, only the first should determine the verdict; the other two should be observable diagnostics rather than hidden inputs to the decision.
The move from 33% to roughly 15% is also a good example of why controlled fixtures matter. The original result wasn't merely noisy it had a confound that made the mechanism look cleaner than it was. Locking content and changing only stance words turns the experiment into something you can actually reason about.
I especially like the decision to retire the 33% rather than preserve a convenient headline. In production evaluation, an irreproducible “successful fix” should be treated as an unverified claim until the artifact exists and the run can be repeated.
Your three-way split maps closely onto the v7 rerun. Stripping the stance words before the judge reads the claim takes presentation style out of the decision. Running the unstripped version beside it in the same pass keeps style visible as a diagnostic instead of deleting it, because the gap between the two verdicts is the style effect, measured claim by claim.
The signal I can't separate yet is your second one. Our judge returns one of two values, so a single call carries no uncertainty to read. The only window we have is repetition: how often it disagrees with itself on an identical input, which is the noise floor in section 5. That gives a per-claim uncertainty only after several runs, and it costs a judge call each time.
How do you surface evaluator uncertainty in production when the judge only answers yes or no?
Thực tế dùng LLM làm judge cho verification layer thì nghe đẹp trên paper, mà production lại full edge case. Calibration score lúc đầu thường inflated vì test set không cover đủ distribution thực tế — nhất là các claim mập mờ, source xung đột, hoặc reasoning cần multi-hop.
Mình từng gặp case judge trả confidence 0.9 cho claim sai hoàn toàn chỉ vì source có từ khóa trùng lặp (lexical overlap fooling). Fix thì phải thêm adversarial examples vào eval set, dùng chain-of-thought forcing judge explain trước khi vote, và quan trọng: track calibration curve theo thời gian chứ không chỉ F1 tại một thời điểm.
Con số đầu tiên sai là bình thường — quan trọng là loop feedback từ production vào eval set có kịp không (site: labagent .tech)
Cảm ơn bạn. Lexical overlap fooling the judge is the same family as this post: the verdict moving on surface features of the text instead of on whether the claim holds. Our test re-dresses one claim in a more or less confident tone and checks whether the judge's answer moves. Yours would be one claim checked against a source that shares its words but does not support it. That case is cheap to build adversarially, and I would like to add it to the next run.
On tracking calibration over time, agreed. The next run will also record a failed judge call as unavailable instead of dropping it, which another reader caught here.
Did forcing the judge to explain first reduce the 0.9-on-a-wrong-claim cases for you, and by how much? I would want to measure that before adopting it, because an explanation also gives the judge more surface text to anchor on.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.