# All rivers run into the sea: Unified Modality Brain-like Emotional Central Mechanism

arXiv: https://arxiv.org/abs/2407.15590 · HTML: https://mlsanity.com/paper/2407.15590/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 51 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2307.02227/) (2023)
- [Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text](https://mlsanity.com/paper/2005.11882/) (2020)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [From Efficient Multimodal Models to World Models: A Survey](https://mlsanity.com/paper/2407.00118/) (2024)
- [Fusing Audio, Textual and Visual Features for Sentiment Analysis of News Videos](https://mlsanity.com/paper/1604.02612/) (2016)
- [Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies](https://mlsanity.com/paper/1712.02896/) (2017)
- [OUS: Scene-Guided Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2405.18769/) (2024)
- [Suppressing Uncertainties in Degradation Estimation for Blind Super-Resolution](https://mlsanity.com/paper/2406.16459/) (2024)

## Built on by (most cited first, 5 of 5)

- [Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotations](https://mlsanity.com/paper/2601.13225/) (2026)
- [Collaborative Adaptive Curriculum for Progressive Knowledge Distillation](https://mlsanity.com/paper/2603.20296/) (2026)
- [Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition](https://mlsanity.com/paper/2604.02397/) (2026)
- [Cognition-Inspired Dual-Stream Semantic Enhancement for Vision-Based Dynamic Emotion Modeling](https://mlsanity.com/paper/2604.12777/) (2026)
- [Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition](https://mlsanity.com/paper/2605.21417/) (2026)

## Abstract (arXiv)

In the field of affective computing, fully leveraging information from a variety of sensory modalities is essential for the comprehensive understanding and processing of human emotions. Inspired by the process through which the human brain handles emotions and the theory of cross-modal plasticity, we propose UMBEnet, a brain-like unified modal affective processing network. The primary design of UMBEnet includes a Dual-Stream (DS) structure that fuses inherent prompts with a Prompt Pool and a Sparse Feature Fusion (SFF) module. The design of the Prompt Pool is aimed at integrating information from different modalities, while inherent prompts are intended to enhance the system's predictive guidance capabilities and effectively manage knowledge related to emotion classification. Moreover, considering the sparsity of effective information across different modalities, the SSF module aims to make full use of all available sensory data through the sparse integration of modality fusion prompts and inherent prompts, maintaining high adaptability and sensitivity to complex emotional states. Extensive experiments on the largest benchmark datasets in the Dynamic Facial Expression Recognition (DFER) field, including DFEW, FERV39k, and MAFW, have proven that UMBEnet consistently outperforms the current state-of-the-art methods. Notably, in scenarios of Modality Missingness and multimodal contexts, UMBEnet significantly surpasses the leading current methods, demonstrating outstanding performance and adaptability in tasks that involve complex emotional understanding with rich multimodal information.