Missing knowledge
ImportError, AttributeError or an unknown keyword. The code calls an API that changed in 2026.
Fix: Nano with live Tavily docsA coding agent that runs every fix against a locked unit test on Nemotron Nano first, and only pays for live docs or a bigger model when the test says why it failed.
Pick a failure type to see which rung of the ladder catches it.
Five routing strategies ran the same ten Python tasks. A patch only counts when the read-only test passes.
of tasks verified by Mjolnir, 10 of 10
per verified fix, 52% below Ultra with Tavily always on
2026 API changes fixed by bare Nemotron Ultra
A bigger model cannot guess a method that was renamed after it was trained. A web search cannot solve a graph algorithm. Mjolnir reads the error and sends each bug down the right road.
ImportError, AttributeError or an unknown keyword. The code calls an API that changed in 2026.
Fix: Nano with live Tavily docs
AssertionError on an edge case. The API is fine; the logic is wrong. Search would only add noise.
Fix: escalate to Super, then Ultratest_solution.py mode 0444 · sha fd6cb989
First, the cheapest model tries the fix against a test it cannot edit.
AttributeError: 'UserPayload' has no attribute 'dict'
Then the traceback is cut to three lines that name the cause.
SchemaGuard 3.0: use .model_dump_strict()
Next, only API errors pull the current docs from the web.
- user.dict(exclude_none=True) + user.model_dump_strict( + exclude_none=True)
Finally, the test passes and you get the diff with its cost.
Six tasks use SDK changes from 2026; four are pure logic. Same tasks, same locked tests, five ways of choosing a model.
| Strategy | How it routes | API changes | Logic | Solved | Cost per fix |
|---|---|---|---|---|---|
| Mjolnir 2D ladder | Nano first, Tavily on API errors, Super or Ultra on logic | 6 / 6 | 4 / 4 | 100% | $0.001671 |
| Ultra + Tavily always | Biggest model, web search on every task | 6 / 6 | 3 / 4 | 90% | $0.003484 |
| Nano + Tavily | Small model with docs, no escalation | 5 / 6 | 1 / 4 | 60% | $0.003396 |
| Ultra alone | Biggest model, no docs | 0 / 6 | 4 / 4 | 40% | $0.002613 |
| Nano alone | Small model, one try | 0 / 6 | 1 / 4 | 10% | $0.000330 |
Ten tasks is a small sample, so read these as a method check, not a market benchmark. Cost per fix divides each strategy's total spend by the tasks it actually solved. Raw receipts: live_5arm_telemetry.json · deterministic_5arm_benchmark.json.
Choose a task to read the diff, the docs Tavily returned and the path up the ladder.
- return user.dict(exclude_none=True)
+ return user.model_dump_strict(exclude_none=True)
Loading the Tavily receipt from the benchmark artifact.
Pick a task and a baseline. The workbench runs both side by side against the same locked test and shows what each one cost.