Model Routing's Hidden Complexity

The world of model routing isn’t as straightforward as it seems. Our experience in building routing into agentic systems has shown that what appears to be a simple problem quickly becomes a complex challenge.

One dimension that makes model routing surprisingly hard is cost. We expected GPT-4.1 to be cheaper than Claude Sonnet 4.6, but our tests revealed the opposite: Sonnet cost $79 total ($0.19/task) while GPT-4.1 cost $155 ($0.37/task), nearly double.

The explanation for this anomaly lies in caching – a factor that’s often overlooked when discussing routing. Agent workloads tend to reuse large chunks of context across steps, leading to high cache hit rates. When these rates are high, effective input costs drop dramatically.

This experience highlights the importance of considering actual cost, which depends on the interaction between the model, workload, and serving infrastructure. A router that only looks at pricing sheets is optimizing against the wrong numbers – a mistake with significant implications for data analysis tools.

Another dimension that complicates model routing is complexity. A common strategy is to estimate task difficulty and send harder tasks to stronger models, but this approach breaks down in two ways: difficulty can be invisible at routing time, and it’s just one signal among many that needs to be balanced with cost, latency, and other factors.

Secondly, even if task difficulty could be perfectly estimated, it’s not the only consideration. In production environments, routers need to balance multiple competing demands, including compliance requirements, data residency rules, privacy constraints, and approved model lists – all of which can influence where a task should go.

We’ve found that building effective routing systems requires considering more than just cost or quality: latency, reliability, and governance are also crucial factors. This complexity is often overlooked in discussions about routing systems, but it’s essential for data analysis tools that need to handle real-world applications with nuance.

Latency is another challenge when it comes to model routing. It’s tempting to think about latency solely in terms of model size – bigger models are slower, smaller ones are faster – but user experience depends on more than just model speed: infrastructure factors like hardware, cache warmth, and endpoint busy-ness often dominate end-to-end response times.

A theoretically faster model can still produce a slower experience if the serving conditions aren’t right. Routing granularity also plays a crucial role in determining latency: routing once per task adds minimal overhead, but routing at every step introduces additional decision points that increase latency and operational complexity.

These lessons have shaped our approach to building routers for agentic systems – we’ve shifted from treating routing as a classification problem to an optimization challenge. Our algorithm optimizes across cost, quality, and latency simultaneously while staying lightweight enough not to become a bottleneck itself.

The figure below illustrates the result on the AppWorld Test Challenge with a CodeAct agent. Each blue square represents a different configuration of our router, tracing out a cost-accuracy frontier – it shows that the router provides a range of operating points for choosing between cost, latency, or accuracy priorities.

Configuration 1 (latency-optimized) achieves 84% accuracy for $93 and 83s: a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% accuracy drop. This is what our experience has led us to conclude about routing – it’s not really about choosing models; it’s about optimizing systems.

Models are just one variable – an important but not sole consideration alongside caching behavior, infrastructure state, compliance constraints, and workload patterns. When routing works well, it’s rarely because it found the ‘best’ model for a given task: it’s because it found the best operating point for the entire system.

That’s a harder problem than classification, but it’s the one worth solving – we believe that optimizing systems will lead to more effective data analysis tools. We’ll be sharing more about our technical approach in a follow-up post and look forward to hearing from others who are building routing into their own agentic systems.

Our experience has also shown us that standard difficulty-based routers often fall short, landing in similar accuracy ranges but at higher cost – they don’t explore the full tradeoff space that an optimization-based approach can. Our algorithm’s lightweight nature (roughly 6 ms and 2 kB of memory per task) means it doesn’t become the bottleneck we warned about earlier.

Our router optimizes across multiple factors, including cost, quality, latency, compliance, and reliability – all while staying adaptable to changing workloads and infrastructure conditions. This is a more complex problem than classification, but one that’s essential for building effective routing systems.

We’re sharing our findings in the hope of helping others who are struggling with model routing challenges – if you have any tradeoffs or complexities you’d like to discuss, we’d love to hear about them.

The complexity of model routing is often overlooked, but it’s a crucial consideration when designing data analysis tools. We believe that by prioritizing system optimization over model selection alone, we can build more effective and efficient routing systems – ones that truly meet the needs of real-world applications.