Repository navigation
fix(agent): a null choice label made the answer undecodable - #508
Merged
NandhaKishorM merged 1 commit intoSep 27, 2026
Merged
Conversation
`67edd5f` rejected a `null` **score level**, on the grounds that a label "is rendered as option
text and used as the answer key". A `choice` label is used for exactly those two things, and the
check was not extended to it. Measured on `main` (`4066d5d`), English checkpoint:
```python
q = {"q": {"type": "choice", "instructions": "which?", "criteria": ["billing", None]}}
agent.system_one("my invoice is wrong", q)["answers"]["q"]
```
```json
{"type": "choice", "choice": "billing",
"probabilities": {"billing": 0.7679, "null": 0.2321},
"confidence": 0.2183, "answer_confidence": 0.7679}
```
Two problems in that one response.
**The option text and the answer key disagree.** `_to_internal` normalises a list of labels to
`{label: None}`, so `render_options` renders the label through `str(k)` — the option the model chose
between was `billing` and the text **`None`** — while the answer key and the `probabilities` key for
that same option are **`"null"`**, because a dict key has to be a `str` and a criteria dict with a
`None` key was apparently not the shape anyone had in mind.
**`"null"` is not decodable.** A client cannot tell whether that option was the *string* `"null"` or
JSON `null`, because both produce that key:
```python
criteria = ["billing", None]
criteria[answer["choice"]] # -> None, but the key said "null"
answer["choice"] == "null" # -> False, so the key is unreachable from the response
```
The answer is therefore not usable for a client that round-trips the key back into the criteria it
sent, which is the documented pattern.
## What it does now
```python
{"criteria": ["billing", None]}
-> ValueError: question 'q': choice label 1 is null; a label is rendered as option text and used
as the answer key, so it must be a string, number or bool -- a null label renders as the text
"None" while its answer key is "null"
```
Over `/v1/systemone` that is a 422, since `serve` maps `ValueError` from the agent to 422.
Deliberate choices tested:
- **Rejected rather than rendered consistently.** The alternative is to keep accepting `null` and make the round-trip work by rendering the option text as `"null"` too. Both are defensible, and the maintainer's own commit rejects the score-level case rather than stringifying it, so this follows the precedent — but it is the one thing here worth a second opinion, because it turns a 200 into a 422 for anyone sending a `null` label today.
- **`""` is still accepted, and that is the point.** An empty-string label already round-trips: `render_options` renders it as `""`, the answer key is `""`, and `criteria[""]` finds it. Pinned in the tests, because "reject the empty-looking labels" would be the wrong generalisation of this fix.
- **Dict-shaped `criteria` is untouched.** `None` as a **value** there is the documented "no description" spelling used throughout `tests/`, and a dict key cannot be `None` anyway, so nothing needed changing on that path.
- **One guard, both backends.** It goes in `Agent._check_question`, which `ONNXAgent._encode_states` already calls (`onnx_agent.py:285`), so the two cannot drift the way the `lang_temperatures` parsing did.
- **A nested label is still a different error.** The existing `unhashable` guard fires first for a list/dict/set label; this one is only about `null`.
## Relationship to the existing checks
`laya.structured` already rejected this shape, one layer up, and says so in different words:
```
test_structured.py:127 [None, "null"] -> SchemaError: enum values produce duplicate choice labels
```
So the collision was recognised for JSON schemas but not for a directly-written question. This
closes the direct path; it does not change the schema path.
Issue NandhaKishorM#302 reports this area. Its point 2 (a null **score level**) was already fixed by `67edd5f`
before this PR, and I have said so on the issue rather than duplicating it; this is the `choice`
half, which that commit left open.
```
$ python tests/test_criteria.py 149 passed, 0 failed (139 before)
$ python tests/test_router.py 546 passed, 0 failed
$ python tests/test_batch.py 49 passed, 0 failed
$ python tests/test_structured.py all structured tests passed
$ python tests/test_onnx_lang_parity.py 11 passed, 0 failed
$ python -m pytest tests/test_serve.py tests/test_router_batch.py \
tests/test_system_one_lang.py tests/test_audit_regressions.py -q
65 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```
All of the above run in CI on every Python the workflow covers, and the whole set is weight-free —
the tests use the existing `_tiny_agent()` in `tests/test_criteria.py`, not a downloaded checkpoint.
Known limitation: a `null` label inside a **dict** `criteria` cannot be tested, because a dict key
cannot be `None` in Python — that path is unreachable rather than merely unchecked.
Owner
|
Thanks @PerryLink. Merged, combined with #496 in |
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 29, 2026
A choice question given its criteria as a list uses each label as an object key and as the answer key. Python rejects a null label (#508): its answer key comes back as the string "null", which a client cannot tell apart from a real "null" label, and `criteria[answer.choice]` never finds it. The TypeScript checkQuestion accepted null and undefined (and a hole in a sparse list, which forEach skipped), so the same undecodable answer key reached the caller. Reject a null, undefined or missing label with a message naming the question and the index, and say "string, number or bool" in the nested-label message, as Python now does. Dict criteria and the empty string are unchanged. This is a parity fix; it makes no prediction-accuracy claim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NandhaKishorM
added a commit
that referenced
this pull request
Sep 29, 2026
…-label fix(ts): reject a null choice label, as Python does (#508 parity)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
A
nullchoicelabel is accepted today, and the answer that comes back cannot be decoded.67edd5frejected anullscore level on the grounds that a label "is rendered as option textand used as the answer key" — a
choicelabel is used for exactly those two things, and the checkwas not extended to it. Measured on
main(4066d5d), English checkpoint, throughsystem_one:{"type": "choice", "choice": "billing", "probabilities": {"billing": 0.7679, "null": 0.2321}, "confidence": 0.2183, "answer_confidence": 0.7679}Two problems in that one response.
The option text and the answer key disagree.
_to_internalnormalises a list of labels to{label: None}, sorender_optionsrenders the label throughstr(k): the option the model chosebetween was
billingand the textNone. The answer key and theprobabilitieskey for thatsame option are
"null", because a dict key has to be astr."null"is not decodable. A client cannot tell whether that option was the string"null"orJSON
null, because both produce the same key:So an answer that includes the null label is not usable by a client that round-trips the key back
into the criteria it sent, which is the documented pattern.
After
{"criteria": ["billing", None]} -> ValueError: question 'q': choice label 1 is null; a label is rendered as option text and used as the answer key, so it must be a string, number or bool -- a null label renders as the text "None" while its answer key is "null"Over
/v1/systemonethat is a 422, sinceservemapsValueErrorfrom the agent to 422.The one judgement call worth your opinion
This is a behaviour change: it turns a 200 into a 422 for anyone sending a
nulllabel today.The alternative is to keep accepting
nulland make the round-trip work instead, by rendering theoption text as
"null"so it matches the key. Both are defensible, and I picked rejection onlybecause
67edd5frejected the score-level case rather than stringifying it — if you would ratherhave the other, this is a two-line change.
""is deliberately not rejected: unlike a null it already round-trips (render_optionsrenders it as
"", the answer key is"", andcriteria[""]finds it), and it is pinned in thetests so the fix is not generalised to "reject the empty-looking labels".
Scope
criteriais untouched. ANonevalue there is the documented "no description" spelling used throughouttests/, and a dict key cannot beNoneanyway.Agent._check_question, whichONNXAgent._encode_statesalready calls (onnx_agent.py:285), so the two cannot drift the way thelang_temperaturesparsing did.Relationship to the existing checks
laya.structuredalready rejected this shape one layer up, in different words:The collision was recognised for JSON schemas but not for a directly-written question. This closes
the direct path and leaves the schema path as it is.
@mandu5's #302 reports this area. Its point 2 (a
nullscore level) was already fixed by67edd5fbefore this PR — I have said so on the issuerather than open a duplicate PR for it — and this is the
choicehalf, which that commit left open.Checks
All of these run in CI, and the whole set is weight-free — the tests use the existing
_tiny_agent()intests/test_criteria.py, not a downloaded checkpoint. Atest (python 3.14)run is not possible for me locally, so the CI matrix is the check for that.