Skip to content

fix(agent): a null choice label made the answer undecodable - #508

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/null-choice-label
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/null-choice-label

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

What this fixes

A null choice label is accepted today, and the answer that comes back cannot be decoded.
67edd5f rejected a null score level on the grounds that a label "is rendered as option text
and used as the answer key" — a choice label is used for exactly those two things, and the check
was not extended to it. Measured on main (4066d5d), English checkpoint, through system_one:

q = {"q": {"type": "choice", "instructions": "which?", "criteria": ["billing", None]}}
agent.system_one("my invoice is wrong", q)["answers"]["q"]
{"type": "choice", "choice": "billing",
 "probabilities": {"billing": 0.7679, "null": 0.2321},
 "confidence": 0.2183, "answer_confidence": 0.7679}

Two problems in that one response.

The option text and the answer key disagree. _to_internal normalises a list of labels to
{label: None}, so render_options renders the label through str(k): the option the model chose
between was billing and the text None. The answer key and the probabilities key for that
same option are "null", because a dict key has to be a str.

"null" is not decodable. A client cannot tell whether that option was the string "null" or
JSON null, because both produce the same key:

criteria = ["billing", None]
criteria[answer["choice"]]   # -> None, but the key said "null"
answer["choice"] == "null"   # -> False, so the key is unreachable from the response

So an answer that includes the null label is not usable by a client that round-trips the key back
into the criteria it sent, which is the documented pattern.

After

{"criteria": ["billing", None]}
-> ValueError: question 'q': choice label 1 is null; a label is rendered as option text and used
   as the answer key, so it must be a string, number or bool -- a null label renders as the text
   "None" while its answer key is "null"

Over /v1/systemone that is a 422, since serve maps ValueError from the agent to 422.

The one judgement call worth your opinion

This is a behaviour change: it turns a 200 into a 422 for anyone sending a null label today.
The alternative is to keep accepting null and make the round-trip work instead, by rendering the
option text as "null" so it matches the key. Both are defensible, and I picked rejection only
because 67edd5f rejected the score-level case rather than stringifying it — if you would rather
have the other, this is a two-line change.

"" is deliberately not rejected: unlike a null it already round-trips (render_options
renders it as "", the answer key is "", and criteria[""] finds it), and it is pinned in the
tests so the fix is not generalised to "reject the empty-looking labels".

Scope

  • Dict-shaped criteria is untouched. A None value there is the documented "no description" spelling used throughout tests/, and a dict key cannot be None anyway.
  • One guard, both backends. It goes in Agent._check_question, which ONNXAgent._encode_states already calls (onnx_agent.py:285), so the two cannot drift the way the lang_temperatures parsing did.
  • A nested label is a different error. The existing guard from fix(serve): a request with no state was answered about the literal text null #427 still fires first for a list/dict/set label.

Relationship to the existing checks

laya.structured already rejected this shape one layer up, in different words:

test_structured.py:127   [None, "null"] -> SchemaError: enum values produce duplicate choice labels

The collision was recognised for JSON schemas but not for a directly-written question. This closes
the direct path and leaves the schema path as it is.

@mandu5's #302 reports this area. Its point 2 (a null score level) was already fixed by
67edd5f before this PR — I have said so on the issue
rather than open a duplicate PR for it — and this is the choice half, which that commit left open.

Checks

python tests/test_criteria.py            149 passed, 0 failed   (139 before)
python tests/test_router.py              546 passed, 0 failed
python tests/test_batch.py                49 passed, 0 failed
python tests/test_structured.py          all structured tests passed
python tests/test_onnx_lang_parity.py     11 passed, 0 failed
python -m pytest tests/test_serve.py tests/test_router_batch.py \
                 tests/test_system_one_lang.py tests/test_audit_regressions.py -q
65 passed
python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
python -m compileall -q laya/ tests/

All of these run in CI, and the whole set is weight-free — the tests use the existing
_tiny_agent() in tests/test_criteria.py, not a downloaded checkpoint. A test (python 3.14)
run is not possible for me locally, so the CI matrix is the check for that.

`67edd5f` rejected a `null` **score level**, on the grounds that a label "is rendered as option
text and used as the answer key". A `choice` label is used for exactly those two things, and the
check was not extended to it. Measured on `main` (`4066d5d`), English checkpoint:

```python
q = {"q": {"type": "choice", "instructions": "which?", "criteria": ["billing", None]}}
agent.system_one("my invoice is wrong", q)["answers"]["q"]
```

```json
{"type": "choice", "choice": "billing",
 "probabilities": {"billing": 0.7679, "null": 0.2321},
 "confidence": 0.2183, "answer_confidence": 0.7679}
```

Two problems in that one response.

**The option text and the answer key disagree.** `_to_internal` normalises a list of labels to
`{label: None}`, so `render_options` renders the label through `str(k)` — the option the model chose
between was `billing` and the text **`None`** — while the answer key and the `probabilities` key for
that same option are **`"null"`**, because a dict key has to be a `str` and a criteria dict with a
`None` key was apparently not the shape anyone had in mind.

**`"null"` is not decodable.** A client cannot tell whether that option was the *string* `"null"` or
JSON `null`, because both produce that key:

```python
criteria = ["billing", None]
criteria[answer["choice"]]        # -> None, but the key said "null"
answer["choice"] == "null"        # -> False, so the key is unreachable from the response
```

The answer is therefore not usable for a client that round-trips the key back into the criteria it
sent, which is the documented pattern.

## What it does now

```python
{"criteria": ["billing", None]}
-> ValueError: question 'q': choice label 1 is null; a label is rendered as option text and used
   as the answer key, so it must be a string, number or bool -- a null label renders as the text
   "None" while its answer key is "null"
```

Over `/v1/systemone` that is a 422, since `serve` maps `ValueError` from the agent to 422.

Deliberate choices tested:

- **Rejected rather than rendered consistently.** The alternative is to keep accepting `null` and make the round-trip work by rendering the option text as `"null"` too. Both are defensible, and the maintainer's own commit rejects the score-level case rather than stringifying it, so this follows the precedent — but it is the one thing here worth a second opinion, because it turns a 200 into a 422 for anyone sending a `null` label today.
- **`""` is still accepted, and that is the point.** An empty-string label already round-trips: `render_options` renders it as `""`, the answer key is `""`, and `criteria[""]` finds it. Pinned in the tests, because "reject the empty-looking labels" would be the wrong generalisation of this fix.
- **Dict-shaped `criteria` is untouched.** `None` as a **value** there is the documented "no description" spelling used throughout `tests/`, and a dict key cannot be `None` anyway, so nothing needed changing on that path.
- **One guard, both backends.** It goes in `Agent._check_question`, which `ONNXAgent._encode_states` already calls (`onnx_agent.py:285`), so the two cannot drift the way the `lang_temperatures` parsing did.
- **A nested label is still a different error.** The existing `unhashable` guard fires first for a list/dict/set label; this one is only about `null`.

## Relationship to the existing checks

`laya.structured` already rejected this shape, one layer up, and says so in different words:

```
test_structured.py:127   [None, "null"] -> SchemaError: enum values produce duplicate choice labels
```

So the collision was recognised for JSON schemas but not for a directly-written question. This
closes the direct path; it does not change the schema path.

Issue NandhaKishorM#302 reports this area. Its point 2 (a null **score level**) was already fixed by `67edd5f`
before this PR, and I have said so on the issue rather than duplicating it; this is the `choice`
half, which that commit left open.

```
$ python tests/test_criteria.py        149 passed, 0 failed   (139 before)
$ python tests/test_router.py          546 passed, 0 failed
$ python tests/test_batch.py            49 passed, 0 failed
$ python tests/test_structured.py      all structured tests passed
$ python tests/test_onnx_lang_parity.py 11 passed, 0 failed
$ python -m pytest tests/test_serve.py tests/test_router_batch.py \
                   tests/test_system_one_lang.py tests/test_audit_regressions.py -q
65 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

All of the above run in CI on every Python the workflow covers, and the whole set is weight-free —
the tests use the existing `_tiny_agent()` in `tests/test_criteria.py`, not a downloaded checkpoint.

Known limitation: a `null` label inside a **dict** `criteria` cannot be tested, because a dict key
cannot be `None` in Python — that path is unreachable rather than merely unchecked.
@NandhaKishorM
NandhaKishorM merged commit a468a3b into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @PerryLink. Merged, combined with #496 in _check_question: a null label is refused first, then repeated or unhashable ones. It ships in 0.3.21.

NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
A choice question given its criteria as a list uses each label as an object key
and as the answer key. Python rejects a null label (#508): its answer key comes
back as the string "null", which a client cannot tell apart from a real "null"
label, and `criteria[answer.choice]` never finds it. The TypeScript checkQuestion
accepted null and undefined (and a hole in a sparse list, which forEach skipped),
so the same undecodable answer key reached the caller.

Reject a null, undefined or missing label with a message naming the question and
the index, and say "string, number or bool" in the nested-label message, as
Python now does. Dict criteria and the empty string are unchanged.

This is a parity fix; it makes no prediction-accuracy claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Sep 29, 2026
…-label

fix(ts): reject a null choice label, as Python does (#508 parity)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants