mlsanityData to 7 Oct 2026

R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning

arXiv 2503.05379 ↗First posted 7 Mar 2025cs.LG cs.CVcited by 99

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

In this work, we present the first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model in the context of emotion recognition, a task where both visual and audio modalities play crucial roles. We leverage RLVR to optimize the Omni model, significantly enhancing its performance in three key aspects: reasoning capability, emotion recognition accuracy, and generalization ability. The introduction of RLVR not only improves the model's overall performance on in-distribution data but also demonstrates superior robustness when evaluated on out-of-distribution datasets. More importantly, the improved reasoning capability enables clear analysis of the contributions of different modalities, particularly visual and audio information, in the emotion recognition process. This provides valuable insights into the optimization of multimodal large language models.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 4 of its 7 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

4 of 7 references.

References · 4

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  3. 2022MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wildcited by 65
  4. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 56; 2026: 43.

99 papers.

Sort:

By area:

  1. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  2. 2025Thyme: Think Beyond Imagescited by 160
  3. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  5. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  8. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  9. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  10. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  11. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  12. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 41
38 more
  1. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  2. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  3. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  4. 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learningcited by 26
  5. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  6. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  7. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 24
  8. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  9. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  10. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  11. 2025HunyuanOCR Technical Reportcited by 18
  12. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 18
  13. 2025Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counselingcited by 17
  14. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  15. 2025ComfyUI-R1: Exploring Reasoning Models for Workflow Generationcited by 15
  16. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  17. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  18. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  19. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  20. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
  21. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  22. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  23. 2025Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMscited by 8
  24. 2025Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Modelscited by 8
  25. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  26. 2025FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPOcited by 6
  27. 2025AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detectioncited by 6
  28. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  29. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  30. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  31. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  32. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Modelscited by 5
  33. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  34. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 5
  35. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  36. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  37. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  38. 2026Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactionscited by 5

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AffectGPT-R1: Leveraging Reinforcement Learning for Open-Vocabulary Multimodal Emotion Recognition
  2. 2026AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition ↗
  3. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
  4. 2025OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs
  5. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
  6. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
  7. 2025Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis
  8. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis