mlsanityData to 7 Oct 2026

From Routing Signals to Selective Review: Visual regrounding in MoE VLMs

arXiv 2609.38111 ↗First posted 29 Sep 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) may accept false visual premises, answering questions about a target object's color, count, location, or state even when it is absent. We call this reliability-critical behavior a target-absence grounding failure. Existing visual-grounding detectors primarily rely on generated responses, hidden states, or uncertainty measures. We present the first framework to leverage internal routing decisions in Mixture-of-Experts (MoE) VLMs to detect target absence before generation and guide selective correction. We extract target-token routing probabilities from Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it, train a separate L2-regularized linear detector for each model, and use its predictions to selectively invoke a target-aware review prompt. Using routing alone, the Qwen and Gemma detectors achieve ROC-AUCs of 0.9988 and 0.9956 on GQA-Inpaint and retain 0.8095 and 0.7781 on the external OBER dataset, respectively. The resulting routing-gated policy improves end-to-end accuracy on GQA-Inpaint and OBER by +22.25% and +12.17% for Qwen, and by +13.42% and +1.39% for Gemma, without modifying model weights. Further analysis shows that the signal is localized to the target-object token, emerges in early MoE layers, and is distributed across partially substitutable experts. Although cross-dataset threshold shifts require recalibration, false-positive review causes limited harm overall, suggesting that intervention risk can be controlled through joint selection of the detector threshold and review prompt. Overall, we show that routing probabilities alone preserve actionable information about visual perception, allowing computation already produced by an MoE VLM to support low-cost detection and selective visual regrounding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 25 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

23 of 25 references.

References · 23

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  6. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  10. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  11. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  12. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
11 more
  1. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  3. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  4. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  5. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  6. 2024Reference-free Hallucination Detection for Large Vision-Language Modelscited by 23
  7. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  8. 2025Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Trainingcited by 16
  9. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 13
  10. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  11. 2026Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination
  2. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
  3. 2026Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding
  4. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations
  5. 2026Visual Grounding Safety in Vision-Language Models
  6. 2026VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought
  7. 2025Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens
  8. 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction