Published June 25, 2026 · Updated July 24, 2026
TL;DR: Open models are now strong enough for real software engineering work, and in this Faros cohort, they challenged expensive frontier defaults directly. Ron Meldiner, Field CTO at Faros, ran 211 real engineering tasks through seven AI coding routes to compare quality, cost, runtime, and consistency on real repos.
Claude Code + GLM-5.2 and Claude Code + Kimi K2.6 landed in the top quality band, while GLM-5.2 was meaningfully cheaper and faster. Claude Code + Opus 4.8 and Codex + GPT-5.5 did not buy their way into the top quality band in this run.
The key takeaway is not to pick a model from public benchmarks alone. Instead, measure routes against your own codebase, workflows, costs, and review standards, then route work based on what actually performs.
Are open source AI models good enough to replace frontier models?
I keep hearing the same question in customer and partner conversations: If open models are getting this good, why are so many software teams still paying frontier-model prices for every AI coding task?
At Faros, we spend a lot of time connecting engineering activity to outcomes: which work shipped, which reviews churned, which CI runs failed, and where teams lost time. Managing and optimizing AI coding spend is now part of that picture. A token bill by itself tells you almost nothing. It’s more useful to know where the spend produced a patch, whether the patch was good enough to move work forward, and whether a cheaper model could have handled the same job.
So I ran Time Machine, our proprietary Faros analysis for evaluating historical engineering work, against our own engineering work in order to answer one question: Where can an open model handle real software development well enough that the expensive route stops being the default?
The idea is simple: Take real tasks and historical PR context, run the same task cohort through selected AI coding harnesses, and compare quality, projected cost, runtime, cache behavior, and invalid/no-diff outcomes.
The short answer surprised me. Claude Code + GLM-5.2 was statistically close to Claude Code + Kimi K2.6 on quality, while being meaningfully cheaper and faster in this cohort. The Opus and Codex routes lagged behind on both quality and economics in this cohort. That gave me a more specific operating rule: test GLM-5.2 and Kimi as the practical default for broad production-like throughput.
The rest of this post walks through what we tested, what changed my mind, and how I would run a similar test inside another engineering organization.
Can you trust public AI benchmarks for choosing a coding model?
The public signal is strong enough that software leaders should pay attention. Open models are closing benchmark gaps quickly, and recently released open models such as GLM-5.2, Kimi K2.7, Qwen3-Coder, and DeepSeek V4 are explicitly aimed at long-context, tool-using, agentic coding workflows.
Those benchmarks set priors. They cannot see your repositories, test setup, review standards, cache behavior, CI failure modes, permission boundaries, or the mix of work your engineers actually need help with. A model can look strong on a leaderboard and still be the wrong default for your environment if the harness wastes context, the cache economics break, or the model struggles on your common task shapes.
I wanted a Faros-specific answer. Which model and harness should we trust for our repositories, our task mix, and our review standards?
This is related to the new wave of model routers, including Sakana's Fugu, which dynamically orchestrates multiple models behind a single API. I like that direction. The limit is that a generic router still needs local evidence: the judge and routing policy have to reflect your codebase, tests, permission boundaries, review burden, cache behavior, and the work slices your engineers actually do.
How we tested different AI coding models on real engineering tasks
For our experiment, we compared model plus harness combinations. I care about the pairing because engineers do not use a raw model in isolation. They use a model through a harness that decides how context is loaded, how tools are called, how diffs are produced, and how failures show up. I wanted the methodology visible before the result, because this experiment is only useful if the reader knows what kind of evidence it is.
| Question | Experiment detail |
|---|---|
| What was the main question the experiment intended to answer? | Could an open-model route handle real Faros engineering work well enough that the frontier route should stop being the default for some tasks? |
| What work source did we use? | Historical Faros engineering tasks and PR context, transformed into Time Machine task rows. (This is local company evidence, not a public benchmark.) |
| How were tasks selected? | The source pool was 500 selected historical Faros PR/Jira instances across 12 repositories. The headline comparison uses the 211 rows that survived the strict cohort filter. |
| Were they sampled randomly, stratified, or handpicked? | The cohort was curated for diversity and operational representativeness. We selected 500 task<>PR pairs from the most active Faros repositories, with coverage across complexity levels and work types: bugs, feature work, KTLO/maintenance, infra/devex, migrations, code-review-style work, and several product surfaces. The selector then balanced within each repo across category and size buckets and filtered out unusable rows, such as missing base commits, missing gold patches, etc. |
| What was the main cohort? |
211 strict instances across 12 Faros repositories. Work type: 61 KTLO/maintenance tasks, 54 feature tasks, 49 infra/devex tasks, 27 bug fixes, and 20 other tasks. Complexity: 81 very-high-complexity tasks, 61 high-complexity tasks, 50 medium-complexity tasks, and 19 low-complexity tasks. Repository domain: 87 UI/reporting tasks, 67 data graph/platform tasks, 47 AI/agent/ML tasks, and 10 connector-ingestion tasks. |
| What routes did we compare? |
OK: OpenCode + Kimi K2.6 — Fireworks default CK: Claude Code + Kimi K2.6 — Fireworks default OG: OpenCode + GLM-5.2 — Fireworks default, Max reasoning CG: Claude Code + GLM-5.2 — Fireworks default, Max reasoning OO: OpenCode + Opus 4.8 — default high effort CO: Claude Code + Opus 4.8 — default high effort Codex: Codex + GPT-5.5 — configured high reasoning |
| Did each harness receive equivalent context and tool permissions? | Each route received the same task setup: the same historical repo state, task/PR context, prompt shape, and permission to produce a patch. What varied was the route itself: the harness/model pairing. So this is not a raw model benchmark; it measures the practical route an engineer would actually use. |
| Were routes run blind? | The implementation routes were blind to the historical solution. Each route received the same scrubbed task context and repo state at the base commit; it did not receive the accepted PR diff, test patch, PR number, merge metadata, author, issue hints, or evaluator-only fields. No route received extra manual help or additional context beyond the shared setup. |
| Did the judge see model identity? | The scoring code used a model-based rubric judge over the task description, generated patch, and per-task rubric. The judge prompt did not include the route, harness, or implementation model identity; those labels were retained separately as experiment metadata/provenance. |
| Was the judge model-based, human-reviewed, test-based, or composite? | The headline judge score was model-based rubric scoring. |
| How were invalid/no-diff cases handled outside the strict cohort? | They were tracked separately in implementation telemetry as invalid, no-diff, failure, or timeout cases. They were not mixed into the headline mean-score table. |
| What was excluded by the strict cohort filter? | Rows were excluded from the headline cohort if any selected seven-way route lacked a valid implementation or judge score. |
| Were costs actual billed costs, projected normalized costs, or replay-estimated costs? | Projected normalized/list costs from token telemetry and provider pricing. |
| Were retries allowed? | Gap-fix and recovery reruns were used where needed to repair telemetry or completion gaps. During the experiment, we found a caching issue that was specific to the Claude Code + Kimi route. We worked with the Fireworks team to resolve it, then reran that route and used the cache-fixed rerun in the final comparison, excluding the earlier low-cache CK run. |
| Were routes run once or multiple times? | The headline table uses one implementation and one score per route/task, not an average over repeated trials. Some selected rows came from rerun or gap-fix sources; the run source is retained in the appendix. |
| What outcomes did we measure? | Aggregate judge score, task win rate, average rank score, projected cost per task, runtime, cache share, generated patch bytes, and invalid/no-diff behavior. |
| What definitions matter for reading the results? |
Time Machine: Faros analysis for replaying historical engineering work. Harness: coding-agent wrapper around the model. Strict cohort: every selected route had a valid implementation and score. Judge score: model-based rubric score. Projected cost: token telemetry plus provider pricing. Provider default: Fireworks-hosted Kimi and GLM rows used provider/model defaults. Cache share: share of input context served from cache. |
| What should the reader keep in mind? | This is a production-style routing evaluation. It is useful for deciding which route to test as a default, but it should not be read as a universal model leaderboard. |
GLM-5.2 vs. Kimi K2.6 vs. Claude Opus: Quality and cost results
GLM-5.2 and Kimi both landed in the top quality band. Claude Code + GLM-5.2 scored 0.568 and Claude Code + Kimi scored 0.566, close enough that I would not treat the difference as a quality win by itself. The operational difference was economics: in this cohort, GLM-5.2 delivered comparable quality while running faster and costing meaningfully less per task. That makes GLM-5.2 the route I would test first for broad production-like throughput, with Kimi still worth keeping in the routing pool where local data shows an edge.
| Variant | Judge score | Cache share | Avg time/task | Avg cost/task |
|---|---|---|---|---|
| CG | 0.568 | 89.7% | 321s | $0.92 |
| CK | 0.566 | 90.0% | 386s | $1.78 |
| OG | 0.556 | 95.5% | 620s | $1.01 |
| CO | 0.521 | 99.7% | 775s | $1.76 |
| OK | 0.514 | 94.8% | 557s | $0.86 |
| Codex | 0.466 | 92.8% | 392s | $2.06 |
| OO | 0.454 | 100.0% | 544s | $1.05 |
Because cache share was nearly identical after the rerun, 89.7% for Claude Code + GLM-5.2 and 90.0% for Claude Code + Kimi, the cost gap was not a cache artifact.
The expensive frontier baselines did not buy their way into the top quality band in this cohort. Claude Code + Opus 4.8 trailed both Claude Code + Kimi and Claude Code + GLM-5.2 on mean score; its projected cost was roughly in the same range as Claude Code + Kimi, but materially higher than Claude Code + GLM-5.2. Codex + GPT-5.5 is included here as a high-reasoning current-default comparison, not as a Fireworks route.
How should you route AI coding tasks by work type?
The aggregate table tells me which routes are efficient overall. The work-type slices tell me how I would route real work. I would not turn the aggregate winner into a one-size-fits-all rule.
Mean score tells me average quality. Average rank tells me which route stayed consistently near the top across tasks, even when it did not win outright.
For this cohort, the routing policy would start here:
| Work type | Start here | Escalate when | Signal from this cohort |
|---|---|---|---|
| Bugs (27) | Claude Code + GLM-5.2 | Claude Code + Kimi for subtle or high-blast-radius fixes | CG had the strongest bug readout: top mean score, top win rate, and top rank. |
| Feature work (54) | Claude Code + GLM-5.2 for routine throughput | Claude Code + Kimi for ambiguity, larger diffs, or review risk | CK led this slice by mean and rank. |
| Infra/devex (49) | Claude Code + Kimi when correctness dominates; OpenCode + GLM-5.2 when consistency matters | OpenCode + Kimi for lower-risk work where cost discipline matters most | Split signal: CK led by mean; OG led by rank. |
| KTLO / maintenance (61) | Claude Code + GLM-5.2 | Claude Code + Kimi when blast radius, dependencies, or review churn are high | CG led by mean. |
| Data graph / platform (67; 40 schema-graph dominant) | OpenCode + GLM-5.2 | Claude Code + Kimi or Claude Code + GLM-5.2 when review risk dominates | OG was strongest, especially on schema-heavy context. |
| UI/reporting, backend/app logic, connector ingestion, tests/CI/docs | Claude Code + GLM-5.2 | Claude Code + Kimi for large or high-review-risk changes | CG had the strongest practical signal across these surfaces. |
| Agent-tooling / AI-heavy (47; 41 agent-tooling dominant) | Claude Code + Kimi for high-risk cases | Claude Code + GLM-5.2 only after local validation on similar tasks | This slice is closer to the hardest work; subtle failures matter more than cost. |
The point is not to memorize these labels. The operating rule is to start with Claude Code + GLM-5.2 where local data says it clears the bar, escalate to Claude Code + Kimi when ambiguity, larger diffs, complexity, or review risk make quality worth paying for, and keep OpenCode + GLM-5.2 as the OpenCode route with the strongest consistency signal.
GLM-5.2 for coding: Why it changed our default model
GLM-5.2 was announced while we were running this experiment, so we added it to the loop. The GLM-5.2 release post emphasized long-horizon engineering, larger context, and stronger coding benchmark performance. That made it worth testing. The completed run made it practical: Claude Code + GLM-5.2 landed in the top quality band, ran faster than Claude Code + Kimi, and cost roughly half as much per task.
This is what I want to emphasize: The best default changed while the experiment was still running. Kimi still remains the route I would keep for quality-sensitive escalation, especially where the local slices show an edge: feature work, very-high-complexity work, larger diffs, and agent-tooling tasks. But GLM-5.2 cleared the bar often enough—and cheaply enough—that it changed what I would test first.
OpenCode + GLM-5.2 also earned its place. It had the best average rank signal, strong cache behavior, and the best data graph / schema-heavy profile. If a team is standardizing around OpenCode, that route deserves serious evaluation rather than being treated as a fallback.
The point is not to pick a winner and freeze it. You need the ability to rerun the evaluation when the market moves, when a provider fixes caching, when a new model ships, or when your own workload changes. This has to become an operating loop and not a one-time bakeoff.
How to run an AI model routing experiment (step-by-step)
Model routing is becoming an important engineering control. If I were doing this experiment with another engineering team, I would start small and make the decision useful within a week. Running your own small experiment could look like this:
| Step | What to measure | Decision it informs |
|---|---|---|
| Pick a representative sample | 20-50 recent tasks across bugs, features, infra/devex, KTLO, and maintenance | Keeps the result grounded in your actual work mix instead of treating a public leaderboard as a proxy for your codebase |
| Run two or three routes | A frontier route, an open-model route, and one alternate harness if available | Shows whether the default should change |
| Score quality and review burden | Patch quality, test impact, reviewer effort, and failure modes | Separates cheap from actually useful |
| Track economics | Projected cost, cache share, runtime, retries, and invalid/no-diff outputs | Finds where savings survive real execution |
| Turn it into a policy | Task types where the cheaper route clears the bar, escalation rules, and rerun cadence | Makes model choice repeatable instead of anecdotal |
Focus your model routing experiment on real tasks like test selection, deploy gates, and review policies. The process should reflect your actual workflows, adapt to changing conditions, and define exactly when to escalate. The goal is not to crown a winning model. The goal is to make model choice repeatable.
Why AI model selection should be a continuous loop
Defaults need evidence. Open models now deserve real engineering evaluations on actual company work, turning model choice into an engineering control like test selection or deploy gates. The market simply moves too quickly for static defaults. When a new model ships, caching improves, or your workload shifts, the right answer changes. Therefore, your most durable capability is continuously measuring your own work to adapt. Model selection must be an operating loop, not a one-time bake-off.
Faros can produce model-routing policies from the engineering systems you already use, drawing on issues, PRs, CI, code ownership, review flow, repo metadata, and delivery outcomes. Time Machine turns that history into a clear guide for which route should handle which work, when to escalate, and where the economics actually survive contact with your codebase. Contact us for a demo today.
Frequently asked questions when comparing open source vs. frontier AI models for coding
Are open source AI models as good as GPT-5.5 or Claude Opus for coding?
Yes, for certain engineering work. In Faros's test of 211 real coding tasks, open models GLM-5.2 and Kimi K2.6 scored in the top quality band, while frontier routes using Claude Opus 4.8 and GPT-5.5 scored lower and cost more. GLM-5.2 matched top quality at roughly half the cost per task.
What is AI model routing?
AI model routing is the practice of directing each engineering task to the model best suited for it, instead of sending every request to one expensive frontier model. Faros found this can cut AI coding costs by roughly 50% at equal quality, because cheaper open models handle routine work while costlier models are reserved for high-risk tasks.
How much can teams save by switching from frontier models to open models for coding?
Roughly 50% per task. In Faros's test, the best open-model route (GLM-5.2) cost $0.92 per task versus $1.76–$2.06 for frontier routes using Claude Opus 4.8 and GPT-5.5—at comparable quality. Actual savings depend on task mix and how much work can safely use the cheaper route.
What are the best open source AI models for coding in 2026?
The leading open source coding models in 2026 include GLM-5.2, Kimi K2.6, Qwen3-Coder, and DeepSeek V4, all built for long-context, agentic workflows. In Faros's testing of 211 tasks, GLM-5.2 and Kimi K2.6 performed strongest on real engineering work, with GLM-5.2 offering the best quality-to-cost ratio.
Can you trust public AI benchmarks when choosing a coding model?
Not fully. Public benchmarks are useful priors but can't account for your codebase, review standards, cache economics, or CI failure modes. Faros's test showed a top-benchmark model can still be the wrong default for a given environment. Validate candidate models against a sample of your own tasks first.
.webp)
.webp)
