mlsanityData to 14 Oct 2026

Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning

arXiv 2601.18321 ↗First posted 26 Jan 2026cs.MM cs.CL cs.CVcited by 0

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Multimodal emotion analysis is shifting from static classification to generative reasoning. Beyond simple label prediction, robust affective reasoning must synthesize fine-grained signals such as facial micro-expressions and prosodic which shifts to decode the latent causality within complex social contexts. However, current Multimodal Large Language Models (MLLMs) face significant limitations in fine-grained perception, primarily due to data scarcity and insufficient cross-modal fusion. As a result, these models often exhibit unimodal dominance which leads to hallucinations in complex multimodal interactions, particularly when visual and acoustic cues are subtle, ambiguous, or even contradictory (e.g., in sarcastic scenery). To address this, we introduce SABER-LLM, a framework designed for robust multimodal reasoning. First, we construct SABER, a large-scale emotion reasoning dataset comprising 600K video clips, annotated with a novel six-dimensional schema that jointly captures audiovisual cues and causal logic. Second, we propose the structured evidence decomposition paradigm, which enforces a "perceive-then-reason" separation between evidence extraction and reasoning to alleviate unimodal dominance. The ability to perceive complex scenes is further reinforced by consistency-aware direct preference optimization, which explicitly encourages alignment among modalities under ambiguous or conflicting perceptual conditions. Experiments on EMER, EmoBench-M, and SABER-Test demonstrate that SABER-LLM significantly outperforms open-source baselines and achieves robustness competitive with closed-source models in decoding complex emotional dynamics. The dataset and model are available at https://github.com/zxzhao0/SABER-LLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 33 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

25 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning2 Aug
  2. 2025Qwen2.5-Omni Technical Report26 Mar
  3. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning17 Jun

Other references · 22

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
10 more
  1. 2024Long Context Transfer from Language to Visioncited by 515
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  5. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  6. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 44
  7. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  8. 2024LearnLM: Improving Gemini for Learningcited by 25
  9. 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
  10. 2025MER 2025: When Affective Computing Meets Large Language Modelscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  2. 2026Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
  3. 2025Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding
  4. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Models
  5. 2025Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding
  6. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding
  7. 2026Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition
  8. 2025Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier