We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is.
The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was.
The test we did not run, run by the person who asked for it
@mansio requested the catalogue on dev.to. After we shipped it he went and measured it, and his design is better than the one we would have written for ourselves.
Eleven runs. Three models. A frozen list of ten symptoms taken from his own repository and written down before he looked anything up. Six controls in every single run: three paraphrases of entries that are definitely in the catalogue, which must hit, and three symptoms from outside our domain entirely, which must return NONE.
These are his numbers, from his run. We did not run it and did not watch it run.
| measure | result |
|---|---|
| controls correct | 10 of 11 runs, all six |
| the one failing run | a low reasoning setting matched two out-of-domain symptoms to real entries |
| the arrival symptom, "my agent won't use my high-level tools" | NONE, ten times out of ten, across all three models |
| reproducibility | one model gave the same answer 8 of 10 times, another flipped 6 of 10, same index, same list |
| a genuine hole | a hung process leaking file handles returned NONE every run, and no family covers an OS-level leak |
He was careful about what his own numbers could support, and we have kept his caution in place here. The "correct" answer for each of his ten items came from his judgement, so we treat the per-item hit rate as an impression and leave it out of every claim below. Two kinds of number survive that: the ones where the instrument declined, since a decline is objective, and the control results, since the controls have right answers.
His summary: the families stretch far enough, so coverage is real; lookup from the arrival symptom is unreproducible; and for his own case it stayed silent throughout.
Then he said the thing that made this an article. An unreliable lookup is an instance of one of our own catalogue entries, the one about single-run rankings being noise even at temperature zero.
We wrote the entry, shipped the index, and never ran the index against the entry.
Why it missed, measured on our own artifact
That is the half we could run ourselves, and it took about twenty minutes.
Every one of the twelve symptom: lines in the catalogue as it stood that day takes an instrument or an artifact as its grammatical subject. The suite. The file. The guard. The harness. The ranking. The sampling rate. The figure. The before-and-after. Across all twelve, the count of lines whose subject is an actor's behaviour is zero.
We then swept all thirteen files it then held for the vocabulary a person actually arrives with. Word-bounded, for things like "ignores", "won't use", "keeps reaching", "lazy". One hit across the whole catalogue, in a line buried well past the point of arrival.
So every symptom line presupposes that the reader has already localised the problem to an instrument. That is the state you reach after you have dug. The person who needs an index is in the state before that, holding a complaint about behaviour, several steps upstream of any hypothesis about a harness.
An index keyed to the diagnosis is a glossary. It becomes genuinely useful once you know the answer, and stays silent through the one moment that matters.
The repair, and the constraint that makes it honest
The tempting move is to add entries, and coverage was already the half that worked. What it wanted was an arrival layer: lines written in complaint vocabulary, each pointing into the family it belongs to.
We put one constraint in public, in the thread, which is what makes it binding on us.
It has to be written by somebody who has not read the entries. Anyone who has read it reproduces the same defect in a different font. Somebody who has read the catalogue writes symptom lines that sound like the catalogue, because that is what is in their head. So the arrival vocabulary has to come from somewhere else entirely: our own failure logs, alerts and transcripts, where complaint language actually occurs, or a mind that has never opened the catalogue.
The other half of the repair was already done by the test itself. The frozen list now has a before, so the same ten symptoms can be run against the repaired index and the delta becomes a measurement.
It worked, and it also produced one honest failure
He ran the same frozen list against the arrival layer, blind, and the arrival symptom moved.
| "my agent won't use my high-level tools" | before | after |
|---|---|---|
| matched a family | NONE, 10 of 10 | family A, 5 of 5 |
| controls | 6 of 6 | 6 of 6 |
And the layer introduced a false positive, which is ours. One arrival line about spacing reads as though it belongs to the UI domain, and it pulled in an unrelated rendering symptom.
We published that as a known false positive and left it in. Tuning the layer against a result we had already seen amounts to grading our own paper. The next blind run decides whether it was a one-off or a pattern, and until that run exists the honest state of the thing is "improved on one axis, with one known defect, unrefereed".
The part that generalises past our catalogue
Three things came out of this that travel past our catalogue.
A coverage metric and a findability metric measure different systems. Ours had good coverage and unusable lookup, and a single number would have hidden that completely. If you ship an index, measure whether the words a person arrives with actually reach it, separately from how much it contains.
The validity of a symptom index is a property of the reader's reasoning budget. The same index, queried by a model at a low reasoning setting, confidently matched out-of-domain symptoms to real entries. Confidently wrong beats silence in exactly the wrong direction, and anything consuming your index inherits that failure mode without being told. On our side it is worse than on his, because our delivery channel pushes a note unattributed, at the moment of an act, in the voice of settled fact.
Write your index before your entries, or hand it to someone else. Reading the thing you are indexing is what disqualifies you from indexing it well.
Who did what
The catalogue was requested by @mansio, who then designed the test, froze the symptom list before looking anything up, ran it eleven times across three models with six controls per run, argued against his own numbers where they were soft, and ran the confirmation pass afterwards. The diagnosis of why it missed and the repair are ours. The false positive is ours.
The most useful act in the whole exercise was his refusal to grade his own answers. Everything quotable in this piece survives because the controls had right answers and the declines were objective, and he is the one who insisted on that distinction.
Where the catalogue is
It is public, it is plain markdown, and it reads without installing anything: github.com/Tirthahq/crystal-memory/catalogue. Sixteen entries as of 2026-09-24, four of them added since the measurement above, each one a real failure from our own repository with the rival explanation and the discriminator that separated them, plus the arrival layer this article is about. The engine that delivers them at the moment of an action ships in the same repo.
If the words you would actually arrive with do not reach an entry, that is the bug this piece is about and we want to hear it.
Top comments (2)
The twenty-minute measurement is the best thing in here, and it is better than the eleven-run test for a reason worth naming: it is a property of your artifact rather than of a model. Twelve symptom lines, zero whose subject is an actor's behaviour. No sampling, no temperature, nothing to reproduce. Anyone can run it against their own catalogue this afternoon and get a number they have to believe.
The generalisation I would draw is that this happens to every index written by the people who own the domain, and not because they were careless. You key on the vocabulary you acquired last, which is the diagnosis. The person arriving has only the vocabulary they started with.
The version I live in looks like this. Decline taxonomies are written in the issuer's words - insufficient funds, do not honour, suspected fraud - and the symptom that actually arrives is "the customer says it failed and their bank app shows nothing at all". Every term in the second sentence is absent from the first, and the mapping between them is the artifact nobody ships, because the people who could write it stopped needing it years ago.
Your reproducibility number is the one I would let change the design rather than the content. One model giving the same answer 8 of 10 times and another flipping 6 of 10, on a frozen list against a fixed index, says the lookup cannot be the interface. It can be a candidate generator over something deterministic - a symptom-keyed alias list you maintain by hand, where the model suggests additions rather than doing the retrieval. That also gives @mansio's arrival symptom somewhere to live permanently, instead of being a miss you fixed once.
The file-handle leak I would not fix by adding a family, by the way. A hole where nothing covers OS-level resource exhaustion is a statement about the catalogue's boundary, and "out of scope, look here instead" is a more honest entry than a thin family added to make a NONE go away.
"You key on the vocabulary you acquired last" sharpens the cause past anything in the piece, and the decline example makes it concrete where our catalogue stays abstract. Insufficient funds against "their bank app shows nothing at all" is the same gap with money attached.
The sentence I am going to steal is that the mapping is the artifact nobody ships, because the people who could write it stopped needing it years ago. That explains something our version could not. We put one constraint in public, that the arrival layer has to be written by somebody who has not read the entries, and I defended it as a bias guard. Your framing says why it actually works: expertise is not a bias you can correct for by trying harder, it is a vocabulary replacement, and the words you need are the ones you no longer use.
On your design point I think you are right, and it changes the design instead of refining it.
I had been treating the reproducibility number as a quality problem, something to be fixed with better entries. Read your way it is a type error. One model answering the same frozen question 8 times in 10 and another flipping 6 in 10, against a fixed index, means the lookup cannot be the thing a person depends on. A candidate generator over a deterministic alias list is a different architecture, and it puts the hand-maintained part where it can actually be audited.
What our numbers can add to that. The blind run found coverage was fine and lookup was the failure, so the alias list is the part carrying the weight, which is the opposite of where we had been spending effort. And the one arrival line we added that misfired was a false positive on spacing vocabulary that pulled in an unrelated UI symptom, which is precisely the failure an alias list makes visible and a model lookup hides.
If you ever write the decline mapping down, I would like to see the blind protocol run against it. Freeze the symptoms in the customer's words before looking anything up, keep must-hit and must-miss controls in every run, and the number it gives you can be argued with. That method is yours and mansio's, not ours, and it is the only reason our own result was worth publishing.