mlsanityData to 7 Oct 2026

Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routing

arXiv 2608.08265 ↗First posted 8 Aug 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Oracle routing measures how much a pool of language models could gain from per-query selection, but the diagnostic has two flaws: testing against a best fixed model selected on the same examples invalidates paired inference, and a full-information oracle sees outcomes no deployable router observes. We separate three estimands (outcome-oracle opportunity, the Bayes-optimal gain from a declared pre-answer signal, and the held-out gain of a learned router) and prove selection-valid confidence intervals that survive choosing the best fixed model or the best member of a router family, a signal-information sandwich, and a (1-1/e) greedy guarantee for building compact pools from submodular complementary coverage. On eight checkpoints from six families over four benchmarks, selection-valid intervals certify a population oracle gap of 9.7--30.7 points on every task, yet the strongest deployable prompt router recovers only 7.5--14.4% of it, and the simultaneous interval for the best of eleven tested policies has lower limit zero throughout. The realizable share of oracle opportunity is small and certifiable: strong routers beat the best fixed model, and most of the gap remains.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

18 of 42 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2017Selective Classification for Deep Neural Networkscited by 799
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  12. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
6 more
  1. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  2. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  3. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 56
  4. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  5. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  6. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agentscited by 0
  2. 2026False Floors: LLM Safety Routing Evaluations Break Under Distribution Shiftcited by 0
  3. 2026Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluationcited by 0