mlsanityData to 14 Oct 2026

What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilings

arXiv 2607.03436 ↗First posted 3 Jul 2026cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Routing benchmarks often compare a policy that commits to one model before seeing its response with a hindsight oracle that credits any correct recorded output. Under stochastic decoding these are different decision classes. For success marginals pim under a frozen query--model--decoder--scorer protocol, we distinguish the clairvoyant single-commit ceiling Ri=maxm pim, product-coupling union Ui^perp=1-prodm(1-pim), and premium Deltai^perp=Ui^perp-Ri. The marginals alone identify exactly the union interval [Ri,min1,summ pim]: zero premium is attainable over compatible couplings, not established for an actual deployment. If generation is independent across models, product is the specified protocol's union probability. We audit frozen correctness tensors from 11 open models and 30 archived responses per query--model cell on GSM8K, MATH-500, and GPQA-Diamond. Full-pool display-channel product premiums are 0.371, 3.542, and 5.100 percentage points; the premium intervals from empirical marginals are [0,0.473], [0,4.787], and [0,7.744]. Their zero lower endpoints are algebraic. All retained scorers, eight finite-draw paths, and all 2,047 nonempty subpools expose scorer, estimator, and pool sensitivity, not confidence intervals. The GSM8K and GPQA display scorers were developed after limited output inspection. A separate retrospective held-out policy illustration instantiates the policy-specific gap decomposition without establishing new-data generalization. A limited reference-based human check supports scorer agreement only on definite-consensus subsets. Hash-bound evidence supports number checks, not end-to-end reproduction. The contribution is a measurement contract for interpreting oracle gaps conditional on coupling, decision class, scorer, pool, and finite draws, not a population effect or an equal-cost routing gain.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

19 of 27 references.

References · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  11. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  12. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
7 more
  1. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  2. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  3. 2024Quantifying Variance in Evaluation Benchmarkscited by 54
  4. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 38
  5. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 13
  6. 2026Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifactscited by 4
  7. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performancecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Most of the LLM Routing Gap Is Task Typecited by 4
  2. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 1
  3. 2026Auditing Proxy-Based Validation Across Text Spanscited by 0
  4. 2026Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluationcited by 0
  5. 2026Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routingcited by 0