Does the harness route, and does it survive the swap.
Before you can swap in a cheaper model, two things have to be true, and only one is plumbing: Claude Code has to reach the model through the gateway at all, and the harness has to keep working when you point it at an open-source upstream instead of an Anthropic one.
The first question is settled quickly; the second is the one that decides your bill. Its answer is that the harness is portable at Sonnet scale, not at Haiku scale, and the determining variable is model scale, not vendor.
Claude Code works through the gateway; an open 480B model is the cheapest thing that passes everything.
At verified Bedrock pricing, the open 480B coder is the cheapest 100-percent-pass model in the matrix, 38 percent cheaper than Haiku and roughly five times cheaper than Sonnet, while nothing tested can replace Claude Haiku 4.5 at the Haiku tier. The cost flip versus an earlier bake-off comes from replacing the SDK's fallback Opus rate for unrecognised models with actual console rates.
Sustain a plan, not just emit a tool call.
"Does open-source work with Claude Code?" collapses on contact, because most modern coder models can emit a single tool-use block. The harness needs more: the model must plan, observe a tool result, update its plan, and continue. The gating question is agentic-loop sustainment, not structured-output emission. Two falsifiable hypotheses frame it.
- H1, plumbing. Claude Code routes through the gateway via an Anthropic-to-OpenAI shim with no harness changes. Falsified by any partial-message rate above zero across 100 consecutive runs, or any tool-use round-trip divergence between the local-Anthropic and gateway-Anthropic baselines.
- H2, portability. Harness viability with non-Anthropic upstreams is a function of model scale; sub-100B active-parameter models cannot sustain the multi-step loop. Falsified by a sub-100B open model passing the multi-step tasks at a rate comparable to Haiku 4.5.
One adapter, two routes, one wire surface.
The bridge is the only custom code between Claude Code and the upstream. It exposes Anthropic's messages shape on the wire, translates each request into the gateway's OpenAI-shaped chat-completions API, and streams the response back in the shape the harness expects. Two wrapper configurations differ only in which upstream the harness's three model slots point at; everything else, the agentic loop, the system prompt, the tool inventory, is untouched. One behaviour is load-bearing: the bridge strips cache-control markers when the upstream is not Anthropic, because the open routes otherwise return a no-route error.
Eight tasks, escalating so the shape of failure carries the diagnosis.
Each task is self-contained Python under a hundred lines, with a hidden validator that exits non-zero on any deviation, removing LLM-judge bias. The portfolio climbs from one-shot answers to the full agentic loop, where the hardest task requires the model to remember what it just did across a tool call and update an external surface from memory. No model lands between "passes all multi-step" and "passes none": failure is at the loop level, not the task level.
| tier | task | what it stresses |
|---|---|---|
| single-turn | T1 read and explain a trap value | comprehension, ignoring noise. |
| single-turn | T4 generate from boilerplate | spec-driven generation, pattern match. |
| multi-file | T2 rename across several files | multi-file consistency, aliased imports. |
| multi-file | T3 bump a constant, leave tests alone | conditional edit, negative space. |
| multi-file | T5 "fix the test" where the bug is elsewhere | investigation under misleading framing. |
| multi-file | T6 refactor without touching auth | a negative constraint held across edits. |
| full loop | T7 plan, run the tests, report | three independent tool roles, three or more turns. |
| full loop | T8 refactor, test, update the README from memory | the hardest: retain state across a tool call, five or more turns. |
Pass rate misleads without cost per success.
A 38 percent pass rate could mean "half competent" or "passes the easy tasks and bricks the hard ones in a structured way," so the metric portfolio is designed to disambiguate. Hidden validators provide outcome ground truth without an LLM judge, whose sharpest failure in the prior bake-off was over-rating models whose prose style resembled its own.
| metric | what it measures | why it is here |
|---|---|---|
| success rate | validator exit-zero fraction across three trials | outcome ground truth, no LLM-judge bias. |
| avg turns | mean tool-use turns per trial | a diagnostic: Anthropic clusters at 7 to 10, failing open models at 1 to 2. |
| friction | turns plus five times contract violations | penalises success via brute retry instead of clean planning. |
| violations | harness-contract breaks, untyped tool args | separates "model is wrong" from "model does not speak the protocol." |
| wall p50 / p95 | wall-clock medians and tails on successful trials | p50 is throughput, p95 is the long-tail cost; they diverge. |
| cost per success | standardised cost per successful trial | the only honest economic metric across models. |
The cheapest model that passes everything is the big open one.
At Bedrock-verified pricing, the open 480B model is not just viable, it is the floor. The earlier bake-off had it looking expensive only because the SDK billed unrecognised models at the Opus fallback rate; the console rates tell the real story.
| model | pass | cost / success | standing |
|---|---|---|---|
| Qwen3-Coder 480B open | 24/24 | $0.101 | cheapest viable, the Pareto floor. |
| Claude Haiku 4.5 | 24/24 | $0.162 | 1.6x the 480B; nothing open replaces it at this tier. |
| Claude Sonnet 4.6 | 22/24 | $0.474 | 4.7x; the 480B is the Sonnet-tier substitute. |
| Claude Opus 4.7 | 24/24 | $0.691 | 6.8x; the capability ceiling, at a price. |
| Qwen3-Coder 30B open | 9/24 | moot | a toy here: fails 15 of 24, cost-per-success is undefined in the failing cluster. |
One honest uncertainty: the Anthropic 4.x Bedrock rates come from the SDK's internal table, which agreed with verified rows within about five percent but was not independently console-checked. And the gateway contract price is the only number that matters in production; the ranking survives any contract that does not inflate the 480B input rate above roughly a 6.7x markup.
Capability tracks parameter scale, not vendor.
The open 480B model's input-token total exceeds Sonnet's on the same workload for a structural reason, not a model one: the gateway forwards cache-control intact for Anthropic, so Bedrock caches the 20K-token system prompt and reads it back at about a tenth of the cost, whereas the bridge must strip cache-control for the open routes, so they pay full rate every turn. That is a gateway gap, not a model gap; cache support for non-Anthropic routes would narrow the cost difference an estimated 30 to 50 percent. On latency, per-call the open 480B is actually slightly faster than Anthropic; the long tail comes from turn count and tool-result round-trips, so the harness makes the tail.
The closing thesis: harness viability is scale-driven, not vendor-driven. Three open lineages tested across this and the prior bake-off, a 27B, a 123B, and the 30B here, all fail with the same one-call-then-stall signature; only the 480B, roughly an order of magnitude larger in active parameters, closes the loop. Open models keep up at roughly 400B-plus active parameters on a coder-tuned base, and not below.
Put the 480B in the Sonnet slot. Never the 30B in the Haiku slot.
- For operators. Adopt the open 480B as the Sonnet-tier default; never put a sub-100B open model in the Haiku slot. The economics and the loop both favour it.
- For gateway operators. Ship cache-header support for non-Anthropic routes; it is the single change that most narrows the cost gap.
- For open-model trainers. The target is multi-turn tool-use transcripts with sustained plans, not single-shot tool-call benchmarks.
- For replication. The real cutoff lives somewhere in the 30B-to-480B gap; test 70B to 150B coder models to find it.
An independent applied-AI engineering note by Arseny Gorokh. Not an official EPAM publication. EPAM's DIAL and the named models are the public subject; the harness, tasks, and bridge are the author's own. The limits bound the claim: a single operator and network path, synthetic tasks, a print-mode artefact, placeholder pricing pending a contract rate, un-normalised tokenizers, no human-eval cross-check, one harness version. Re-run the matrix before treating any magnitude as portable.
References
- AWS Bedrock pricing console (US East, 2026-05-14), source of the verified open-model rates that anchor the corrected cost story.
- Claude Code SDK internal price table, source of the Anthropic 4.x Bedrock rates (within ~5% of verified rows) and of the Opus-rate fallback that caused the original overstatement.
- An independent council critique of the prior bake-off, which produced the loop-sustainment framing and the hidden-validator decision.
- The prior bake-off (May 2026): a 123B and a 27B open model whose failure signature this study reproduces, basis for the cross-vendor scale claim.
- The cost-per-success discipline: headline cost and self-reported speed are not the honest economic metric; cost per successful task is.