# MUPA2E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

arXiv: https://arxiv.org/abs/2608.15999 · HTML: https://mlsanity.com/paper/2608.15999/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 17 of 17)

- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS](https://mlsanity.com/paper/2407.19809/) (2024)
- [Efficient Pain Recognition via Respiration Signals: A Single Cross-Attention Transformer Multi-Window Fusion Pipeline](https://mlsanity.com/paper/2507.21886/) (2025)
- [A Lightweight Transformer for Pain Recognition from Brain Activity](https://mlsanity.com/paper/2604.16491/) (2026)
- [A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities](https://mlsanity.com/paper/2607.19716/) (2026)
- [Explainable graph attention network for stress recognition (StressGAT) via differential action units](https://mlsanity.com/paper/2607.20819/) (2026)
- [Towards a Unified Modality-Agnostic Multimodal Framework for Cognitive Workload Assessment](https://mlsanity.com/paper/2609.20199/) (2026)
- [Combining Facial Videos and Biosignals for Stress Estimation During Driving](https://mlsanity.com/paper/2601.04376/) (2026)
- [Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech](https://mlsanity.com/paper/2604.11417/) (2026)
- [Beyond the Raw Waveform: Fusing Visual Representations of EDA for Stress Detection](https://mlsanity.com/paper/2609.22095/) (2026)
- [A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2608.09088/) (2026)
- [Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition](https://mlsanity.com/paper/2602.20530/) (2026)
- [Efficient and Interpretable Body-Based Emotion Recognition with Lightweight Temporal Convolutional Networks](https://mlsanity.com/paper/2607.20820/) (2026)

## Abstract (arXiv)

Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPAtextsuperscript2E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~30 achieves the highest validation performance and a test accuracy of 70.07%. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of 20 seconds yielded a test accuracy of 62.71%, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.