mlsanityData to 7 Oct 2026

Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

arXiv 2605.21980 ↗First posted 21 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Large Vision-Language Models (LVLMs) represent a significant leap towards empathetic agents, demonstrating remarkable capabilities in emotion understanding. However, the internal mechanisms governing how LVLMs translate abstract visual stimuli into coherent emotional narratives remain largely unexplored, primarily due to the scarcity of visual counterfactuals and the diffuse nature of emotional expression. In this paper, we bridge this gap by introducing a steering-vector-based causal attribution framework tailored for descriptive emotional reasoning. To this end, we construct a specialized dataset to demystify the emotional circuits underlying the three-stage ``Adapt-Aggregate-Execute'' mechanism. Crucially, we discover a functional decoupling: visual emotional cues are aggregated in middle layers via sentiment-specific attention heads, but are subsequently translated into narrative generation in deep layers through emotion-general pathways. Guided by these insights, we regulate the emotional information routing to strengthen attention flow and amplify the semantic activation to consolidate expression. Extensive experiments on the comprehensive MER-UniBench demonstrate that our methods significantly improve performance via inference-time intervention, effectively mitigating emotional hallucinations and corroborating the causal fidelity of the discovered circuits.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 68 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

27 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning7 Mar
  2. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models27 Jan

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  7. 2021Towards Emotional Support Dialog Systemscited by 294
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  10. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  11. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  12. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
13 more
  1. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  3. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  4. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  5. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 22
  6. 2024OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognitioncited by 17
  7. 2025AffectGPT-R1: Leveraging Reinforcement Learning for Open-Vocabulary Multimodal Emotion Recognitioncited by 15
  8. 2024Vision-Language Models Create Cross-Modal Task Representationscited by 14
  9. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  10. 2025Emotion-Qwen: A Unified Framework for Emotion and Vision Understandingcited by 10
  11. 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 5
  12. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  13. 2026Weight Patching: Toward Source-Level Mechanistic Localization in LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
  2. 2025Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models
  3. 2026Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
  4. 2025EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis
  5. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  6. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
  7. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Models
  8. 2026NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control