fix(structured): expose answer_confidence on DecisionResult

[Bruce-Yii]

The user decision

Which of my structured decisions are safe to automate?

You run laya.decide(..., return_details=True), get a DecisionResult per state, and filter it to
decide what routes automatically and what goes to a human. That filter is the decision. Here is the
shipped example from docs/structured.md, verbatim:

result = agent.decide(state, schema=Ticket, return_details=True)
if result.confidence["department"] < 0.6:
    result.values["department"] = "human-review"

It gates on result.confidence.

The failure mode

DecisionResult.confidence is a different quantity. It is built from the answer's confidence field, and
laya/common.py:confidence_from_probs documents that field as:

"Normalized Shannon entropy confidence: 1 - H(p) / log(k). How concentrated the whole distribution
is. Useful, but not calibrated: it is not what temperature scaling fits and not what the reported
ECE measures. See answer_confidence."

The two are also on different scales, which tests/test_confidence.py pins: a two-option
distribution comes back as 0.90 on a noul and 0.53 on an equivalent choice, and the same
documented threshold separates them at 0.85. Issue #394 is open on exactly this — "a confidence
threshold does not transfer across option counts" — which is the entropy definition's specific
failure mode, because log(k) is in the denominator.

So the number the example filters on moves with the shape of the question, not with how right the
answer is. Add a fourth option to department and every field's reported confidence shifts, for
reasons that have nothing to do with the decision.

And confidence defaults to 0.0. structured.py reads answer.get("confidence", 0.0), so a
field that reported no confidence at all, a field the model did not return, and a field whose
confidence really is zero are one value. A caller filtering "below 0.6, escalate" escalates all
three identically and cannot say which it was looking at.

Meanwhile Laya's own abstention gate is defined against the other number. On #361 the maintainer
converged on:

"an opt-in threshold, off by default, that reads answer_confidence (max(p)), because that is
the value the calibration figures describe and it does not drift with the number of options"

So the structured bridge is the one surface reporting a different quantity from the gate that
min_confidence implements, while the docs pointed the reader at the other one.

Maintainer precedent

The closest precedent is #621, where he was asked to change what an existing property reported and
declined — while naming the shape he would take:

"Changing what dtype returns would affect the autocast call itself, so I'd rather not do that.
I'd take a PR that either documents dtype as the autocast target … or adds a small read-only
way to ask what precision a call with N rows runs in."

And #302, which settles that an existing field's meaning is not available to be repurposed:

"Laya's confidence is defined as 1 minus normalised entropy, and the thresholds in Laya's docs are
calibrated to that definition, so I'll keep it rather than change its meaning for existing users."

This PR is exactly that shape: additive, read-only, and confidence keeps its value and its default.
Also #533/#535, where three integrations were given one shared error class so they could not
drift — here DecisionResult and the gate read the reported quantity through one helper.

Boundary inference, with the falsifier. High confidence that this fits: it adds no policy, chooses
no threshold, owns no queue, executes nothing, and leaves every application-owned field alone. The
falsifier is docs/staged-adoption.md:16-18, which lists what a shadow record should contain — and
every item on it (the incumbent action, the reviewed outcome, the review decision) is something Laya
does not know. This changes only the half Laya does know, in an artifact Laya already returns, so it
is not a claim on the record. If the maintainer reads DecisionResult as part of the application-owned
handoff, this should have been a docs correction only, and I would rather be told that than be wrong.

On the name. answer_confidence is the quantity temperature scaling fits and every
calibration figure here is computed on. It is not a claim that the number is right: reading it
as "about c of the answers returned at c are correct" holds only after temperatures are
fitted and validated on held-out data for that checkpoint and question shape, and the shipped
checkpoints are over-confident as shipped. So the reason to report it is not that it is more
trustworthy -- it is that it is the same number the gate and the eval harness use, which is
checkable, rather than a property that has to be taken on trust.

The Laya contract

DecisionResult reports max(p) under its own name, and a field that reported no
usable confidence is None — a value that is not 0.0, because "nobody told me" and "they told me
zero" are different facts and a caller acting on them should not have to guess which one it is.

• DecisionResult.answer_confidence — the max(p) per field, None when absent/unusable.
• DecisionResult.confidence — unchanged: the entropy value, unchanged default of 0.0.
• Read through laya.confidence.answer_confidence_value, which deliberately does not fall back to
  confidence. Reporting the entropy number under this name would be worse than reporting
  nothing, because the name is what the caller filters on. The gate's own read path now calls the same
  helper, so the artifact and min_confidence cannot disagree.

Implementation

• laya/confidence.py — answer_confidence_value(answer); the gate's private _gate_confidence now
  delegates to it and keeps its own entropy fallback. flag_low_confidence is untouched: same
  signature, same semantics, same 0.0 no-op, still public API with a test pinning that.
• laya/structured.py — the answer_confidence field, populated in _details.
• The field is appended with a default, so every existing construction of the dataclass keeps
  working and the first six fields keep their positions. tests/test_structured_api.py pins that
  alongside its field-list check.
• docs/structured.md — shows both numbers, states both definitions, and gates on the one the gate uses.
  That example was the bug report, so fixing it is part of the fix.

Validation

RED → GREEN, weight-free, against 9d95567:

tests/test_structured.py       114 passed, 0 failed   (was 103)
tests/test_structured_api.py    38 passed, 0 failed

The new assertions do not merely check that a field exists. They build a case where the two
quantities rank the same two fields in opposite order, so the test fails if either number is
wrong, and they pin that absent / True / NaN all land on None rather than 0.0.

Full ci.yml suite list — all 62 suites, extracted from the workflow rather than hand-picked:

62 suites, 5 failures -- all five fail identically on unmodified 9d95567 in this
environment (verified by stashing and re-running each):
  test_docker_entrypoint.py  needs docker
  test_env_docs.py           the LAYA_MAX_CONCURRENT doc gap
  test_mcp.py                needs the mcp extra
  test_onnx_quantize.py      needs the onnx extra
  test_tokenizer_cache.py    needs symlink privilege (WinError 1314)

Maintainer gate: ruff, compileall, git diff --check, zensical build --strict --clean — all clean.

Independent of #664 and #679. Both the wrong-quantity problem and the 0.0-means-absent problem
exist on 9d95567, and nothing here reads a key that either of those PRs adds. This can merge in any
order.

Collision

No open PR touches DecisionResult or the confidence fields. Two open PRs do touch the same
files, though, so this is worth stating rather than glossing: #683 and #666 both change
laya/structured.py and tests/test_structured.py (and tests/test_structured_docs.py, which
this one does not touch). They are about the score-minimum projection in _project; this is
about DecisionResult and _details. Different functions, so a merge should be clean in
either order, but the files overlap. #657, #644, #667, #476 and #419 touch none of these files.

Non-goals

• confidence is not changed, redefined, or deprecated. It has a real use — "how concentrated is
  this distribution" — and /v1/systemone: 128-option choices get 422, a null score level comes back as a null legend value, and confidence differs from TypeSafe's formula #302 settled its meaning. Two names, two definitions, both reported.
• No gate, no threshold, no policy. min_confidence still reads the answer's own
  answer_confidence; nothing here makes Laya choose one.
• No change to the values projection. An abstained field is still None in values; changing
  the schema-shaped output would break its contract.
• No change to decide's signature, laya-ts, or any surface other than DecisionResult. feat(ts): port confidence-based abstention gating on answer_confidence (#361) #644
  and fix(ts): honour score minimum in structured projection (#663) #676 are working in laya-ts/src/structured.ts; there is a parity question there, and I would
  rather raise it in review than bundle a second language port into a contract fix.

Tradeoffs

A second number in the artifact. Two confidences where there was one is more to understand. I
think it is right — they are different quantities with different scales and the existing one has a
documented use — but a maintainer who wants a single field would reasonably disagree.

None is new in a Dict[str, float]. Callers iterating answer_confidence must handle a missing
entry as None rather than assume a float. That is the point, and it is also a new thing to
document.

answer_confidence_value is not exported from laya. It is public in laya.confidence and used by
the library; I kept it off the top-level __all__ because test_packaging.py requires every exported
name to be documented, and an exported helper nobody needs to call is API surface for nothing. Say the
word if you would rather it were top-level.

What would falsify this

• If DecisionResult is application-owned in the maintainer's view, this should have been a
  documentation correction and the field is scope creep.
• If reporting both numbers is considered worse than fixing the one example, then the fix is the
  docs/structured.md change alone and I would withdraw the field.
• If nobody filters a DecisionResult on confidence in practice, the field is a nicety. The argument
  is that it is the documented way to decide what to automate, and it currently points at the wrong
  number — but I have not measured how many projects do this, and I have no evidence that they do.

[Bruce-Yii]

CI note: every check on this PR passes except two, and neither is this change.

dependency CVEs — the repo-wide pip-audit --strict audit. It cannot resolve the PyTorch CPU index's 2.14.0+cpu, a local version absent from PyPI. It fails identically on #664, #679, #661, #653, and on unrelated PRs that touch nothing.

smoke (ubuntu-latest) — a PyPI network timeout, not a dependency conflict. The log is five ReadTimeoutErrors reaching pypi.org while resolving click, then ResolutionImpossible because the index was never reachable:

WARNING: Retrying (total=4 ...) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='"'"'pypi.org'"'"', port=443): Read timed out. (read timeout=15)")': /simple/click/
...
ERROR: Cannot install laya because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/

smoke (ubuntu-24.04-arm) and spark-build passed in the same run and build the same image, and tests passed on py3.10 / 3.11 / 3.12 / 3.13 / windows. This diff touches no dependency or build file — docs/structured.md, laya/confidence.py, laya/structured.py, tests/test_structured.py, tests/test_structured_api.py — so there is no dependency change for it to conflict with.

I do not have admin rights on this repository, so I could not re-run that job to demonstrate it green. If it matters before review, the smoke workflow can be re-triggered on demand.

Green: tests (5 matrices), evals, docs (strict build), lint, CodeQL, onnx export, smoke (ubuntu-24.04-arm), spark-build, http, package, Chinese benchmark audit, no unsafe deserialization, secret scan.

[Bruce-Yii]

Amendment (acb2bef): the first commit of this branch said "calibrated confidence", "calibrated max(p)", "how much should I trust this answer", and named the helper calibrated_confidence. All four overstate what the number is, and the README is explicit that they should:

Both checkpoints are over-confident as shipped and laya-multilingual has no fitted temperatures at all, so fit them before relying on these numbers ... Confidence orders decisions; it does not establish that a decision is correct.

answer_confidence is the quantity temperature scaling fits and every calibration figure in this repository is computed on. Reading it as "about c of the answers returned at c are correct" holds only after temperatures have been fitted and validated on held-out data for that checkpoint and question shape. Naming a public helper after a property the shipped checkpoints do not have reintroduces exactly the ambiguity #419 caught — the quantity calibration targets is not the same claim as a calibrated model.

What changed:

• calibrated_confidence(answer) → answer_confidence_value(answer). A name that says which field it reads and asserts nothing about it. Still not in the top-level __all__.
• Every docstring and comment that said "calibrated" now says what the number is (max(p), the probability mass on the reported answer; the quantity temperature scaling fits and ECE evaluates) and states the condition under which it means c≈correctness, including that the shipped checkpoints are over-confident. DecisionResult's docstring, laya/confidence.py, docs/structured.md and the tests all say the same thing.
• The product argument is now stated more precisely, and better: answer_confidence is the number to filter on because it is the same quantity min_confidence compares against and the eval/calibration stack measures — a checkable claim, since the gate and this field read one helper — rather than because it is more trustworthy.
• docs/structured.md no longer implies a calibration guarantee, and its example says explicitly that filtering on the right number is not the same as it being a trustworthy probability, linking the README's Calibration and Honest limits sections.
• Fixed a dangling [Calibration](#calibration) anchor that tests/test_packaging.py caught — that section lives in the README, so it is a cross-doc link now.

Unchanged: the field, the None semantics, the no-fallback rule, and confidence's value and default. flag_low_confidence still has the same signature, semantics and 0.0 no-op.

Re-verified after the amendment: test_structured.py 114, test_structured_api.py 38, test_confidence.py 59, test_packaging.py 106, test_structured_docs.py 87, test_doc_tables.py 109, test_hooks_api.py 387, test_hooks.py 204, test_langchain.py 207. Full ci.yml list, 62 suites, the same 5 failures that fail identically on unmodified 9d95567 here. ruff, compileall, git diff --check and zensical build --strict --clean clean.

[NandhaKishorM]

Passed checks and merged. Thank you @Bruce-Yii! It ships in 0.3.22.