mlsanityData to 14 Oct 2026

Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?

arXiv 2609.32996 ↗First posted 26 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Policies trained from the same base model can appear to solve different problems. An oracle that chooses the best policy for each problem may therefore appear much stronger than any single policy. Selecting the largest estimated success rate also selects favorable sampling errors. We study this effect through reliability coverage, the fraction of problems whose success probability reaches a chosen threshold. Our first test redistributes stored correctness outcomes across policies within each problem. A second also preserves each policy's total successes, accounting for overall quality differences under a specified statistical model. For five training seeds of a seven-billion-parameter vision-language model, redistribution reproduces 0.096 of an estimated 0.113 oracle gap at threshold 0.10. Neither test finds significant evidence at this threshold. Small advantages remain unresolved. Mixtures, routers, voting, and weight averaging show no detectable improvement over their corresponding single-policy baselines. Training policies on different datasets shows little detectable specialization under light post-training, and no router gain. A stronger recipe does create it, both tests detect it, and a router gain appears only at the high thresholds where the specialists separate. In a control with predictable specialization, a router recovers about half the oracle gap. Coverage bounds explain why even a genuine oracle advantage need not yield a deployment gain. The tests assess apparent specialization from stored responses before investment in routing. Code: https://github.com/KurbanIntelligenceLab/oracle-gaps

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

25 of 35 references.

References · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
13 more
  1. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  2. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  3. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  4. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  5. 2023A Unified Theory of Diversity in Ensemble Learningcited by 71
  6. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  7. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 54
  8. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 38
  9. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 13
  10. 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundariescited by 9
  11. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  12. 2026Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Liftcited by 5
  13. 2026Sparsity Curse: Understanding RLVR Model Parameter Space from Model Mergingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch
  2. 2026Available Guardrails: Certifying Selective Prediction across ML Systems
  3. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
  4. 2026Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
  5. 2023More for Less: Safe Policy Improvement With Stronger Performance Guarantees
  6. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning
  7. 2022Offline Policy Comparison with Confidence: Benchmarks and Baselines
  8. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention