AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes
On the map: LLM reasoning → Audio-language model reasoning
Abstract (arXiv)
Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding, yet affective reasoning remains vulnerable to shortcut behavior. Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language, which weakens traceability and external verification. Prior reinforcement learning approaches mainly reward context or logical coherence without explicitly enforcing attention to human evidence. In addition, LLM as a Judge scoring often suffers from score clustering, which reduces reward discriminability. We propose AffectOmni, a GRPO trained framework for verifiable affective reasoning. AffectOmni introduces People Focus and Temporal Order rewards to encourage people-centric evidence selection and temporally structured reasoning, and it adopts within-group comparative scoring to produce more stable and discriminative reward signals. For verification, a Thinking Summarizer converts free form rationales into executable evidence instructions, which are grounded into pixel level evidence regions via SAM3 to provide an externally auditable interface outside the training loop. Experiments on IntentBench, Daily Omni, and WorldSense show consistent improvements over open source 7B scale baselines, including gains of 4.66% on emotion recognition and +14.29% on temporally sensitive tasks. Code is available at https://github.com/eliot127825-rgb/AffectOmni_nobody.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 71 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
39 of 71 references.
Closest ancestors · 5
Same direction, newest first
- 2025SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models19 Sep
- 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities23 May
- 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning7 May
- 2025Qwen2.5-Omni Technical Report26 Mar
- 2025Application of Multiple Chain-of-Thought in Contrastive Reasoning for Implicit Sentiment Analysis10 Mar
Other references · 34
- 2023GPT-4 Technical Reportcited by 18,396
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024GPT-4o System Cardcited by 4,588
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2024OpenAI o1 System Cardcited by 1,244
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
- 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
22 more
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
- 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
- 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
- 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
- 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
- 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 87
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
- 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
- 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 49
- 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 43
- 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 41
- 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
- 2023Factify 2: A Multimodal Fake News and Satire News Datasetcited by 23
- 2023Explainable Multimodal Emotion Recognitioncited by 19
- 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 13
- 2025Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learningcited by 13
- 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
- 2025APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimizationcited by 9
- 2025Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewardscited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
- 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
- 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning
- 2026Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
- 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Models
- 2026Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition
- 2023Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations
- 2026Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models