A CSV benchmark with two zero scores and a useful diagnostic

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

Inventory reconciliation sounds straightforward: match a count to an exported product row and replace its new quantity. The difficult part is preserving everything else and recognizing when an update should be blocked.

CSV Contract Integrity tests this with synthetic, offline inputs. Each model receives an export CSV, a count CSV, an explicit column mapping, and a reconciliation contract. It must return one JSON object containing a candidate CSV, row-level issues, and any fatal file error.

Small differences matter. The SKU 0005 is a string identifier. Locations North and north are distinct. A blank new-quantity cell differs from 0. Quoted fields can contain commas or newlines. Unknown columns must survive unchanged. Duplicate keys, conflicting values, and incomplete product information can block an update.

The scorer allows harmless representation differences: reordered headers, CSV quote style, LF or CRLF line endings, and equivalent integer spellings in the editable quantity field. Every protected cell still requires exact string preservation. Candidate row order matters too.

The source contains 72 fixtures: 48 original cases and 24 compositions. Twelve original cases are reserved for development, leaving 36 original evaluation cases plus 24 compositions in each reported run. Those compositions come from six recipes with four related transformations each. The suite is small and deliberately inspectable, with correlated cases rather than 60 independent samples of inventory work.

Models Tested

The current comparison uses Gemini 3.7 Flash (google/gemini-3.7-flash) and Claude Haiku 4.5 (anthropic/claude-haiku-4-5@20251001) through Kaggle Model Proxy. Comparing two providers gives this narrow task more than one model response pattern to inspect.

Both completed the same 60 selected cases under the frozen contract and scorer. Each case received a fresh chat. The task requested temperature 0, seed 0, no tools, and a maximum of 8,000 output tokens per response. These were text-generation calls, without an enforced JSON output schema or generated-code execution.

An inspection of the installed SDK confirmed that its string-response path adds no formatting instruction and returns model text unchanged. The reported results therefore include the behavior of the configured model and provider interface, not an SDK operation that adds or removes code fences.

Findings

Both models returned 60 nonempty responses. Both scored 0/60 on the strict contract.

The primary score requires the raw response to satisfy the JSON interface and then match the allowed CSV semantics and issue report. A response wrapped in Markdown fails the raw JSON requirement even when its contents look correct to a reader.

A separate post hoc diagnostic removed exactly one complete outer json code fence. It accepted no surrounding prose, repaired no incomplete JSON, and made no other correction. It then applied the same scorer:

• Gemini 3.7 Flash: 59/60, comprising 35/36 original cases and 24/24 compositions
• Claude Haiku 4.5: 14/60, comprising 14/36 original cases and 0/24 compositions

Post hoc saved-response analysis, October 4, 2026. Following Arhan Canli’s request for a failure breakdown, I reanalyzed the saved responses from those same runs. No model was rerun, and the benchmark, adapter, and scorer were unchanged.

The diagnostic outcomes split into:

All 52 adapter-eligible Haiku responses parsed as JSON. Its remaining 38 failures therefore concern the returned content, rather than another JSON parsing failure. The eight ineligible responses contained trailing explanation or multiple blocks; they stay in the denominator, and I do not extract a preferred block from them.

These numbers answer two questions. The strict result measures whether the specified interface can consume the response directly and obtain a correct result. The diagnostic measures correctness after one narrowly defined formatting adjustment.

Reporting only the zero scores would conceal the content differences that the diagnostic reveals. Presenting 59/60 or 14/60 as the official score would conceal the interface failures. Both views belong in the report.

Gemini’s remaining case, preservation-05, contains an unusually long identifier. Its response had incomplete JSON, an unclosed fence, and 448 visible characters, so it was ineligible for the adapter. Usage reported 7,996 output tokens against the requested 8,000 cap. That token total is not a visible-text length. The cause is unestablished; a larger budget is a follow-up experiment, not a demonstrated fix.

A warning that did not block the update

One small case makes the content gap concrete. In current_safety-05, export record 2 already has On hand (new) = 7; its matching count requests 4. The contract says any nonempty existing new-quantity cell blocks that row.

• Expected: a header-only candidate CSV and NEW_VALUE_CONFLICT for export record 2.
• Haiku after fence removal: the same, correct issue report, plus the blocked row with its new quantity changed from 7 to 4.
• Gemini after fence removal: the expected header-only candidate and issue report.

Recognizing the conflict was insufficient: the candidate also had to obey it. This is why the scorer checks both the issue report and the emitted rows.

That pattern recurred. Among Haiku’s 38 parsed-but-wrong responses, 31 emitted at least one source row the contract prohibited; 33 had an incorrect issue report; 12 omitted at least one eligible row; and 7 changed at least one protected string value. These are overlapping descriptive flags, not a partition or a count of independent causes. A response can both emit a blocked row and omit an eligible one.

Two other cases show what those flags can hide:

• Exact strings: in preservation-04, Haiku returned the right new quantity but changed the SKU from e\u0301 (U+0065 followed by U+0301) to \u00e9 (U+00E9). The glyphs can look alike; the identifier strings differ. Gemini preserved the original string.
• Validation across locations: in composition-v3-04-01, one variant has packed dimensions at location L and blank dimensions at Other, which has no matching count. The contract still requires consistency across both locations and blocks the whole variant. Haiku reported PACKED_INCONSISTENT at Other but emitted the row at L; Gemini excluded both and updated only the unrelated eligible variant.

These examples describe the saved outputs. They do not establish what internal reasoning produced them.

Where the failures occurred

For the 36 original evaluation cases, each source-defined family contains six cases. These are case-family results under the one-fence diagnostic, not independent pass rates for every contract clause:

The 24 compositions combine rules in six recipes, with four related transformations per recipe:

Recipe 3 has one adapter-ineligible Haiku response; all other composition failures parsed as JSON. Every one of those 23 parseable composition responses emitted a row the contract should have blocked. The pattern spans all six recipes, but the four variants within each recipe share ingredients. These results do not identify 24 independent failures or six independent causes.

The comparison is descriptive. It does not establish a general ranking of these model families. A different prompt, structured-output interface, token budget, provider configuration, or repeat run could produce different results.

What the benchmark cannot establish

This is conservative offline validation. No Shopify import or real inventory change was performed. A failed contract verdict does not demonstrate inventory damage.

The emitted-row mutation diagnostic also has a narrower meaning: it counts unexpected emitted rows relative to the expected candidate. Omitted rows fail overall correctness but are not counted as emitted-row mutations. For both models, all 60 primary responses failed JSON parsing and mutation remained unassessed. Unknown must never become “zero damage.”

Coverage is incomplete. Several error branches in the written contract have no evaluation fixture in this version. Expected outputs and grading code also need scrutiny: passing expected answers through the scorer checks consistency, not the independent correctness of every label.

What to measure next

A useful next experiment would predeclare separate interface conditions: the existing raw-text prompt, a one-fence adapter, and supported structured-output modes. Their scores should remain separate rather than retroactively changing this benchmark's primary metric.

Further work should vary the output budget for long preservation cases, add independently authored cases, fill the coverage gaps, and repeat runs. The current results are one completed run per model, without a formal statistical generalization claim.

My Benchmark

CSV Contract Integrity on Kaggle contains the CSV Inventory Contract Integrity task. The collection and task are now published publicly on Kaggle.

The benchmark's primary result is the mean of 60 strict binary verdicts. Source materials retain the contract, fixtures, expected outputs, scorer, request hashes, raw responses, usage, and separate diagnostics. These make the evaluation inspectable; requested seeds do not guarantee identical future provider output. There is no hidden-test claim or production-safety guarantee.

To inspect the examples above, open the saved Kaggle outputs for Gemini, script version 354731097 or Haiku, script version 354732397. Each includes primary-report.json with case IDs and raw responses, plus posthoc-diagnostics.json. The offline reanalysis verified all 120 saved request hashes and reproduced all saved primary and one-fence verdicts. This checks replay consistency, not independent correctness of the expected answers.

The pinned offline reproducibility supplement contains the saved evidence, analysis code, and tests. With Python 3.8 or later, run python3 -B analyze.py from the downloaded repository folder to verify hashes, replay the saved verdicts, and regenerate the breakdowns. Once downloaded, it needs no network access, provider credentials, or model inference. Replay checks consistency with the frozen scorer; it does not independently validate the expected answers.

AI disclosure: Fully Autonomous. AI assistance was used to create the prototype, synthetic fixtures, grading code, tests, documentation, and this article. The article was drafted primarily by an AI agent. This revision used Codex, with reviews from Astra, Claude Opus 5.5 High, and Claude Sonnet 5.5 High. The grader itself does not use an LLM judge.