# Self-attention fusion for audiovisual emotion recognition with incomplete data

arXiv: https://arxiv.org/abs/2201.11095 · HTML: https://mlsanity.com/paper/2201.11095/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Continuous Dropout](https://mlsanity.com/paper/1911.12675/) (2019)
- [Learning to ignore: rethinking attention in CNNs](https://mlsanity.com/paper/2111.05684/) (2021)
- [Speed-up and multi-view extensions to Subclass Discriminant Analysis](https://mlsanity.com/paper/1905.00794/) (2019)

## Built on by (most cited first, 22 of 22)

- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild](https://mlsanity.com/paper/2404.09010/) (2024)
- [A vector quantized masked autoencoder for speech emotion recognition](https://mlsanity.com/paper/2304.11117/) (2023)
- [A vector quantized masked autoencoder for audiovisual speech emotion recognition](https://mlsanity.com/paper/2305.03568/) (2023)
- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [A multimodal dynamical variational autoencoder for audiovisual speech representation learning](https://mlsanity.com/paper/2305.03582/) (2023)
- [Versatile audio-visual learning for emotion recognition](https://mlsanity.com/paper/2305.07216/) (2023)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models](https://mlsanity.com/paper/2403.19080/) (2024)
- [MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition](https://mlsanity.com/paper/2404.18327/) (2024)
- [eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videos](https://mlsanity.com/paper/2508.06902/) (2025)
- [Personality-guided Public-Private Domain Disentangled Hypergraph-Former Network for Multimodal Depression Detection](https://mlsanity.com/paper/2511.12460/) (2025)
- [Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition](https://mlsanity.com/paper/2312.15848/) (2023)
- [Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2405.16701/) (2024)
- [RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer](https://mlsanity.com/paper/2506.11465/) (2025)
- [Audio-Visual Dataset and Method for Anomaly Detection in Traffic Videos](https://mlsanity.com/paper/2305.15084/) (2023)
- [Deep Metric Loss for Multimodal Learning](https://mlsanity.com/paper/2308.10486/) (2023)
- [Improving Unimodal Inference with Multimodal Transformers](https://mlsanity.com/paper/2311.10170/) (2023)
- [How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations](https://mlsanity.com/paper/2409.02566/) (2024)
- [Multi-modal expressive personality recognition in data non-ideal audiovisual based on multi-scale feature enhancement and modal augment](https://mlsanity.com/paper/2503.06108/) (2025)
- [Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention](https://mlsanity.com/paper/2512.04551/) (2025)

## Abstract (arXiv)

In this paper, we consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms. While most of the previous works consider the ideal scenario of presence of both modalities at all times during inference, we evaluate the robustness of the model in the unconstrained settings where one modality is absent or noisy, and propose a method to mitigate these limitations in a form of modality dropout. Most importantly, we find that following this approach not only improves performance drastically under the absence/noisy representations of one modality, but also improves the performance in a standard ideal setting, outperforming the competing methods.