mlsanityData to 14 Oct 2026

A Regime Theory of Controller Class Selection for LLM Action Decisions

arXiv 2605.06339 ↗First posted 7 May 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Deployed language and vision-language models must decide, on each input, whether to answer directly, retrieve evidence, defer to a stronger model, or abstain. Contrary to the common monotonicity intuition, greater per-input expressivity is not uniformly beneficial in finite samples: under identical strict cross-validation, different benchmarks prefer different controller classes. This reflects a finite-sample limitation of instance-level uncertainty signals, which can be exhausted at a distribution-dependent scale. We organize controllers into a nested lattice of four classes: fixed actions, partition routers, instance-level controllers, and prior-gated controllers, ordered by complexity. We prove a regime theory that turns three data-estimable bottlenecks into a class choice: how much improvement is possible beyond the best fixed action, whether there are enough samples for instance-level controllers to make reliable decisions, and how much improvement a coarse partition router can recover when instance-level signal is unreliable. The resulting Bernstein-tight threshold has a matching information-theoretic lower bound, and strict nested cross-validation provably selects a near-best class. Across SMS-Spam, HallusionBench, A-OKVQA, and FOLIO, the predicted class matches the empirical winner; the prior-gated controller wins on TextVQA when OCR tokens supply a label-free prediction-time prior. Code is available at https://github.com/Anonymous-Awesome-Submissions/Regime-Theory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 43 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

28 of 43 references.

References · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2017Selective Classification for Deep Neural Networkscited by 852
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
16 more
  1. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  2. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  3. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  4. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  6. 2023Calibrated Language Models Must Hallucinatecited by 115
  7. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  8. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  9. 2021Optimal strategies for reject option classifierscited by 61
  10. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  11. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  12. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
  13. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 37
  14. 2025(Im)possibility of Automated Hallucination Detection in Large Language Modelscited by 15
  15. 2022AUC-based Selective Classificationcited by 13
  16. 2024Hierarchical Selective Classificationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Models
  2. 2026Training Language Models to Cooperate with Inference-Time Controllers
  3. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
  4. 2026REFLEX with Jev for Efficient Selective Control in LLM Agents
  5. 2026Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
  6. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
  7. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
  8. 2026Sparsity Is Necessary: Polynomial-Time Stability for Agentic LLMs in Large Action Spaces