S1MB English v1

This is the public evaluation dataset for S1MB (System One Mosaic Benchmark), which evaluates System One Decision Models on Choice, Noul, and Score tasks. It is a test-only collection with 137 benchmarks across 106 active subsets, 14,009 cases, and 26,269 judgments.

Release article · Leaderboard · Evaluation code · Evaluation results

The collection includes tasks derived from existing datasets and six synthetic benchmarks authored and self-reviewed using GPT-6-Astra. The synthetic labels are not independently human-validated. S1MB does not establish unseen-task generalization or training-data non-overlap.

Loading and storage

from datasets import load_dataset

dataset = load_dataset("hotchpotch/S1MB-english-v1", "laya__typed_decisions")
test = dataset["test"]

This repository was renamed from hotchpotch/s1mb-dataset. Historical measurements retain their original Dataset repo IDs and exact commit SHAs; do not rewrite their provenance to use the new name. Pin a Dataset revision when reproducing an evaluation.

Select a configuration explicitly. Parquet uses Zstandard compression and dictionary encoding; all decoded values and row order match the local Arrow release. Only training-manifest.json defines active membership. Parquet files listed in quarantine-manifest.json are retained for audit, excluded from named configurations, and must not enter evaluation. Source attribution and license scope are retained in sources.json, SOURCES.md, dataset_metadata.json and _sources/. README has one leading YAML block listing every config and split. Hub manifests point to Parquet files; _sources/local-arrow/ preserves the historical Arrow manifests.

Typed decisions over shared state, with Choice, Noul and numeric Score supervision.

Use training-manifest.json for the active test membership. all-data-manifest.json also includes quarantined sources for audit only.

Columns

Score values come from the explicit numeric criteria, not candidate indices. Noul retains authored false/true definitions. Preserve soft targets and candidate order. Repeated strings use physical dictionary encoding; decoded values remain strings.

Sources

See sources.json, SOURCES.md and dataset_metadata.json for upstream datasets, revisions, attribution and license scope. The laya__ namespace denotes the Laya-source conversion pipeline; it does not imply Laya authored those upstream datasets. laya__typed_decisions contains four synthetic workflows published by LocalLLaMA. No blanket relicensing is applied. namespace-map.json records names; identifiers remain stable.

This is a test-only benchmark. The table lists cases and judgments in the active training-manifest.json; a case can contain multiple judgments. Quarantined subsets are excluded. Source links include acquisition datasets and original dataset references where recorded.

Licenses below describe the original datasets. Follow the links for their full terms. Unknown means the data license has not been confirmed.

License

Each subset follows its original dataset's license and usage terms. Check the linked original sources before using or redistributing the data. This collection does not change those licenses.

Where noted, code, annotations and source documents have different terms. Detailed source records are in sources.json. An unknown license is not permission to reuse the data.

For Open-Jev, retain its acquisition attribution and third-party notices. In open_jev__customer-control-v1, the generated conversations have a CC0-1.0 declaration, but the upstream question descriptions have no verified license and are not relicensed as CC0. Authored S1MB generalization subsets retain their recorded not separately declared status. See sources.json, SOURCES.md and the retained upstream notices for scope and qualifications.

Evaluation and result submission

S1MB supports English (english-v1) and Japanese (japanese-v1) evaluation. This repository supplies English inputs and targets; Japanese evaluation inputs and targets remain private; both languages publish measured results to hotchpotch/s1mb-result. Run s1mb run --category english-v1 for this scope or s1mb run --all with authorized access to both datasets. Follow the evaluation guide.

Keep targets and provenance out of inference text. Fix evaluation settings before running; do not tune parameters or select configurations using test labels or measured scores. Submit validated measurements to hotchpotch/s1mb-result, following the submission guide.