Proving routed code is as good: a verification playbook
A cost cut that shows up later as bugs isn't a saving. Here's how to check routed work before your engineers depend on it.
Routing only pays if the routed work is still right. The good news is that the jobs worth routing (reading, searching, boilerplate) are also the easiest to check. The discipline is in checking them on your code, before rollout, and continuing to check as models change.
Build the test set from your own backlog
Public benchmarks are useful for comparing models in general, but they don't reflect your languages, frameworks, and conventions. Task-based benchmarks such as SWE-bench, which evaluates models on real GitHub issues checked against the repository's own tests, show the right shape: real tasks, real tests. Apply the same idea to recent tickets and pull requests from your own repositories.
Run both ways
- Run each task routed and unrouted, under the same conditions.
- Check both against your tests, builds, and linters.
- Have engineers spot-check a sample of diffs blind to which route produced them.
- Record cost, time, and outcome for every run.
Gate per job, not per model
A cheap model might be excellent at reading and summarizing files but weak at writing tests in your framework. Set quality gates per job type, keep each job on the cheap route only where it clears the bar, and send the rest back to the frontier model.
References
- Jimenez, C. E. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.