# From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing

arXiv: https://arxiv.org/abs/2606.06924 · HTML: https://mlsanity.com/paper/2606.06924/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 27 of 27)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute](https://mlsanity.com/paper/2506.22716/) (2025)
- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [TensorOpera Router: A Multi-Model Router for Efficient LLM Inference](https://mlsanity.com/paper/2408.12320/) (2024)
- [EcoAssistant: Using LLM Assistant More Affordably and Accurately](https://mlsanity.com/paper/2310.03046/) (2023)
- [Eagle: Efficient Training-Free Router for Multi-LLM Inference](https://mlsanity.com/paper/2409.15518/) (2024)
- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing](https://mlsanity.com/paper/2508.12631/) (2025)
- [Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection](https://mlsanity.com/paper/2405.02134/) (2024)
- [Causal LLM Routing: End-to-End Regret Minimization from Observational Data](https://mlsanity.com/paper/2505.16037/) (2025)
- [Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Tasks](https://mlsanity.com/paper/2305.08714/) (2023)
- [MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing](https://mlsanity.com/paper/2601.17814/) (2026)
- [Life-Cycle Routing Vulnerabilities of LLM Router](https://mlsanity.com/paper/2503.08704/) (2025)
- [How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities](https://mlsanity.com/paper/2504.07113/) (2025)
- [Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers](https://mlsanity.com/paper/2505.12601/) (2025)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://mlsanity.com/paper/2508.00234/) (2025)
- [Dynamic LLM Routing and Selection based on User Preferences: Balancing Performance, Cost, and Ethics](https://mlsanity.com/paper/2502.16696/) (2025)

## Built on by (most cited first, 3 of 3)

- [RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing](https://mlsanity.com/paper/2606.18774/) (2026)
- [Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing](https://mlsanity.com/paper/2609.37362/) (2026)
- [Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing](https://mlsanity.com/paper/2609.37402/) (2026)

## Abstract (arXiv)

Existing LLM routing methods often construct supervision from a single sampled response for each query--model pair. Because LLM generation is stochastic, however, such an observation can be an unstable estimate of model capability: semantically equivalent query formulations and repeated decoding may yield different scores and even different model preferences. We show that this instability can further propagate from routing labels to learned routing policies. To address this issue, we propose DARS (Distribution-Aware Routing Supervision), which estimates query-level model capability from repeated observations spanning semantics-preserving query rewrites and stochastic decoding. DARS summarizes expected quality, expected cost, and performance variability to construct risk-aware supervision without changing the downstream router architecture. Experiments across diverse tasks and routing methods show that DARS generally improves routing utility and cost--quality trade-offs over single-shot supervision. Further analyses show that its benefits persist under moderate sampling budgets and different decoding temperatures. These results suggest that reliable LLM routing should move beyond individual sampled outcomes and instead model query-level capability distributions.