Same prompt, four models: what Opus, Sonnet, Astra and Sol each got wrong

I ran one prompt through four models last week. Opus 5.5 and Sonnet 5.5 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex. The task: a single HTML file where a procedural octopus escapes a glass jar by unscrewing the lid from the inside.

Astra was done in 14 minutes. Sonnet took 72. Opus hit the most bugs of the four and still finished with nothing open, while Sonnet's session had zero recorded bugs and its final build had console warnings.

It's one run per model, so please don't read this as a benchmark. I watched four sessions end to end and changed my prompt because of them. That's all this is.

The task

The prompt was long. This is the line that made it hard:

The octopus must actually unscrew the lid. Not a canned animation: the lid's rotation must be driven by the tentacles' contact with it.

Any model can make an octopus wave near a lid. Here the lid only turns if tentacles grip it, so they have to grab, twist, let go, grab again, and the lid has to climb the thread as it turns. If that logic is wrong, a screenshot usually still looks fine.

Everything else was set dressing. Dim room, wooden table, water in the jar, eight tentacles with suckers, no external models, orbit controls, a pause button and a readout of the phase and the lid angle. And "a single self-contained HTML file", which comes back later.

The results

The times include each agent testing its own work. The Codex runs were at high effort, and the Claude Code runs didn't record one. Different tools, one run each.

GPT-6 Astra, 14 minutes

Astra put everything in one file and treated the lid as a body that can only rotate. Grip forces from the tentacles turn it, and the thread pitch turns rotation into height.

Before writing code, it looked up how similar builds had gone. One of those was my Opus run from below, which didn't find its worst bug until it could step through the animation phase by phase. Eleven minutes in, Astra built itself the same kind of check: a local server plus timed checkpoints which replay the sequence and read the simulation state out of the DOM.

The checkpoints paid off two minutes later. Over the 3.5 turns, the tentacles were encircling the octopus's own body instead of letting go and grabbing a fresh spot within reach. Watching it, you'd see busy arms.

It rewrote how each arm picks its anchor, ran the checkpoints again and was done at minute 14.

Opus 5.5, 25 minutes

Opus built the scene in parts and glued them into a 1,735-line file. The first thing that broke was the skin shader, which wouldn't compile because a variable was called patch. That's a reserved word in GLSL. Rename, move on.

What happened next is why I'm writing this. Instead of waiting for the animation to play out in real time, Opus added temporary hooks so it could jump to any phase and check the numbers. The first jump showed the lid sometimes turning backwards while being unscrewed. I would not have caught that by watching it once.

Once that was fixed, it stepped through the rest: the lid coming off, falling on the table, the octopus squeezing through the neck. It measured 0.68 ms of simulation and 1.1 ms of rendering per frame and deleted the hooks before handing over the file.

GPT-6.1 Sol, 29 minutes

Sol started by picking a palette (brass, walnut, teal glass, a coral octopus) and then wrote a 309-line file where friction between tentacles and lid drives the thread. From minute 16, it read the live state off the canvas, so phase, lid angle, torque and grips were all visible to it.

No bugs recorded. Most of the remaining time went into the look. Light colour, wood grain, light again, then two passes on how the body squeezes through the neck. Its own note at the end says it should have been told the fps target and the lighting mood up front.

The result is solid. The lid comes off at 1260°, bounces on the table, and the octopus ends up next to the jar without clipping through the glass.

Sonnet 5.5, 72 minutes

Sonnet split the work into 14 files and wrote a little build script to concatenate them. It opened a bare stub in the browser at minute 34, before the octopus existed.

It then went deep on things nobody else touched. It profiled uneven frame pacing, found the lid never quite came to rest after falling, stepped through the bounces, and even checked the scene on a 375×812 phone viewport.

No failure was recorded in that session. It did run into a duplicate JavaScript declaration twice, though, and its last check still showed that declaration and some WebGL feedback-loop warnings in the console. Everything rendered and the readout worked, so it shipped like that. It also never ran the whole sequence live without stepping, or on an integrated GPU.

What I'm taking from this

I stopped trusting the bug count. Opus logged two and ended clean because it found both. Sonnet logged none and ended with warnings, because nothing forced it to look at them. The number tells you what got caught.

Every real bug in these four runs turned up while an agent was poking at the running scene. Nobody found one by reading their own code back. Astra and Opus built that kind of tooling early and found their bugs early. Sonnet opened a browser early too, but its checks were mostly about performance, and the warnings got through.

The "self-contained" thing annoyed me more than it should have. Both Claude Code runs pulled Three.js from a CDN anyway (r170 for Opus, r167 for Sonnet), and Opus never opened its file straight from disk. I didn't check the Codex files for this. Either way, if a word matters to you, spell out what it means.

And whatever the prompt left vague is where each run spent its extra time. For Sol that was lighting. For Sonnet it was frame pacing and the falling lid. Neither was wrong to care; I just would have picked something else.

So this goes into the first prompt next time. Every one of the four sessions points to at least one of these lines:

Before you build the scene, build a way to check it:
expose phase, lid angle and grip state on window,
and let me fast-forward to each phase on demand.
"Self-contained" means no network: inline three.js.
Test the file opened from disk, not only from a server.
Target: 60 fps on an integrated GPU.
Lighting: one warm side light, dim room. Do not retune it.

My guess is that with these lines, all four would finish faster and closer together. I'll rerun it and see.

The sessions

All four are public, with the moments that counted pulled out and the raw transcript a click away. I keep them on coders.talk, which I'm building:

• GPT-6 Astra, 14 min
• Opus 5.5, 25 min
• GPT-6.1 Sol, 29 min
• Sonnet 5.5, 72 min

Has a "clean" run ever turned out to be your worst one? And for something physics-heavy like this, which model would you hand it to first?