Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
also in 2 more
Abstract (arXiv)
Vision-Language-Action (VLA) models have shown strong generalization in robotic manipulation by leveraging rich representations from pretrained vision-language models. However, their deployment in real-world environments remains limited by recurring unreliable behaviors. In this work, we study state hallucination, a recurring failure pattern in which a VLA continues acting as if an unrealized robot-object state had been achieved. Our analyses find that state hallucination coincides with weakened attention to task-relevant visual regions, and a mechanistic interpretation via sparse autoencoders reveals that hallucination-associated sparse features are activated when these failures occur. Based on this analysis, we propose SOUL (Sparse feature pOlicy UnLearning), which selectively unlearns policy knowledge associated with state hallucination behaviors, where sparse features identified from hallucination failures and successful behaviors serve as explicit forgetting and retention targets, respectively. Experiments across VLA architectures in simulated and real-world environments show that our method substantially reduces hallucinated failures and improves task success without substantially compromising the existing manipulation capabilities. These results suggest that interpretable feature analysis provides a practical basis for selectively modifying undesirable knowledge in robot policies.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 36 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
11 of 36 references.
References · 11
- 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
- 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
- 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
- 2013k-Sparse Autoencoderscited by 209
- 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
- 2024BatchTopK Sparse Autoencoderscited by 126
- 2025Gemini Robotics: Bringing AI into the Physical Worldcited by 77
- 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 33
- 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulationcited by 5
- 2026Action Hallucination in Generative Vision-Language-Action Modelscited by 5
- 2026Self-Correcting VLA: Online Action Refinement via Sparse World Imaginationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Feedback-Enhanced Hallucination-Resistant Vision-Language Model for Real-Time Scene Understanding
- 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models
- 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
- 2026ROT: Rotating Hidden States towards Contextual Vectors for Hallucination Mitigation in LVLMs
- 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
- 2026VLA-Forget: Vision-Language-Action Unlearning for Embodied Foundation Models
- 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
- 2026Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration