Whose turn is it? Context-guided model routing for stateful agents

arXiv 2610.19292 ↗First posted 13 Oct 2026cs.AIcited by 0

On the map: LLM agents and tool use → Tool learning and function calling also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat requests as independent. Stateful agents do not operate under this assumption: a user turn may require multiple model and tool calls, its difficulty depends on the state produced by earlier turns, and changing models can sacrifice reuse of the conversational prompt cache. We study zero-shot, cost-aware routing at the user-turn boundary, systematically varying the routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior. We evaluate on multi-turn versions of SheetCopilot and SpreadsheetBench 2, where persistent workbooks and cross-turn dependencies make isolated routing decisions especially brittle. At a matched offloading rate, giving the router session history improves task pass rate over the same router without history, with gains for every model pair; history-aware routing remains within one point of always using the expensive model. We also found that cost depends on execution dynamics rather than token prices alone: the cheaper model's per-task cost advantage narrows from 6.8x to 3.2x on longer sessions. A closed form cache analysis further shows that downward switches (expensive to cheap) recover their cache cost within two LLM calls, whereas upward switches do not break even under the evaluated conditions. Together, these results show that agent routing is a sequential decision problem: effective policies must account for what happened earlier in the session and for the asymmetric cost of switching models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 19 of its 28 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

19 of 28 references.

References · 19

2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 111
2025Universal Model Routing for Efficient LLM Inferencecited by 86
2024A Unified Approach to Routing and Cascading for LLMscited by 80
2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 57

7 more

2026When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scalingcited by 16
2026SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflowscited by 11
2026MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddingscited by 10
2026LLMs Get Lost in Evolving User Intentcited by 10
2026From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialoguecited by 4
2026Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Trainingcited by 1
2026HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Poolscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way