# R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.05379 · HTML: https://mlsanity.com/paper/2503.05379/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Multimodal LLMs / vision-language models / [Multimodal emotion understanding](https://mlsanity.com/mllm/multimodal-emotion-understanding/)

## Builds on (most cited first, 4 of 4)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding](https://mlsanity.com/paper/2501.15111/) (2025)

## Built on by (most cited first, 30 of 101)

- [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.06958/) (2025)
- [Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://mlsanity.com/paper/2503.15558/) (2025)
- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [DeepEyesV2: Toward Agentic Multimodal Model](https://mlsanity.com/paper/2511.05271/) (2025)
- [VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning](https://mlsanity.com/paper/2505.12434/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning](https://mlsanity.com/paper/2505.02835/) (2025)
- [VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception](https://mlsanity.com/paper/2509.21100/) (2025)
- [HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context](https://mlsanity.com/paper/2506.21277/) (2025)
- [GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning](https://mlsanity.com/paper/2506.16141/) (2025)
- [ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning](https://mlsanity.com/paper/2505.21457/) (2025)
- [Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration](https://mlsanity.com/paper/2505.20256/) (2025)
- [VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking](https://mlsanity.com/paper/2506.01725/) (2025)
- [EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.04623/) (2025)
- [LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs](https://mlsanity.com/paper/2506.21862/) (2025)
- [LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning](https://mlsanity.com/paper/2509.24786/) (2025)
- [Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1](https://mlsanity.com/paper/2503.24376/) (2025)
- [MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection](https://mlsanity.com/paper/2503.18132/) (2025)
- [Mind with Eyes: from Language Reasoning to Multimodal Reasoning](https://mlsanity.com/paper/2503.18071/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2512.06810/) (2025)
- [HunyuanOCR Technical Report](https://mlsanity.com/paper/2511.19575/) (2025)
- [Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey](https://mlsanity.com/paper/2509.24322/) (2025)
- [Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling](https://mlsanity.com/paper/2505.15715/) (2025)
- [MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models](https://mlsanity.com/paper/2508.09210/) (2025)
- [ComfyUI-R1: Exploring Reasoning Models for Workflow Generation](https://mlsanity.com/paper/2506.09790/) (2025)
- [BaseReward: A Strong Baseline for Multimodal Reward Model](https://mlsanity.com/paper/2509.16127/) (2025)
- [TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs](https://mlsanity.com/paper/2509.18056/) (2025)
- [Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2505.18536/) (2025)

## Abstract (arXiv)

In this work, we present the first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model in the context of emotion recognition, a task where both visual and audio modalities play crucial roles. We leverage RLVR to optimize the Omni model, significantly enhancing its performance in three key aspects: reasoning capability, emotion recognition accuracy, and generalization ability. The introduction of RLVR not only improves the model's overall performance on in-distribution data but also demonstrates superior robustness when evaluated on out-of-distribution datasets. More importantly, the improved reasoning capability enables clear analysis of the contributions of different modalities, particularly visual and audio information, in the emotion recognition process. This provides valuable insights into the optimization of multimodal large language models.