# Multimodal End-to-End Sparse Model for Emotion Recognition

arXiv: https://arxiv.org/abs/2103.09666 · HTML: https://mlsanity.com/paper/2103.09666/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/1905.05812/) (2019)
- [Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis](https://mlsanity.com/paper/1609.05244/) (2016)
- [Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition](https://mlsanity.com/paper/2009.09629/) (2020)
- [Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-Task Learning for Offensive Language Detection](https://mlsanity.com/paper/2004.13432/) (2020)
- [EmoGraph: Capturing Emotion Correlations using Graph Networks](https://mlsanity.com/paper/2008.09378/) (2020)
- [Combating Human Trafficking with Deep Multimodal Models](https://mlsanity.com/paper/1705.02735/) (2017)

## Built on by (most cited first, 30 of 40)

- [Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2310.05338/) (2023)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [M-SENA: An Integrated Platform for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2203.12441/) (2022)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023)
- [A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition](https://mlsanity.com/paper/2111.02172/) (2021)
- [CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition](https://mlsanity.com/paper/2201.03804/) (2022)
- [General Debiasing for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2307.10511/) (2023)
- [Greenformer: Factorization Toolkit for Efficient Deep Neural Networks](https://mlsanity.com/paper/2109.06762/) (2021)
- [Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors](https://mlsanity.com/paper/2203.00257/) (2022)
- [Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss](https://mlsanity.com/paper/2207.11482/) (2022)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)
- [Learning without Isolation: Pathway Protection for Continual Learning](https://mlsanity.com/paper/2505.18568/) (2025)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding](https://mlsanity.com/paper/2112.12180/) (2021)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [Discourse-Aware Soft Prompting for Text Generation](https://mlsanity.com/paper/2112.05717/) (2021)
- [Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition](https://mlsanity.com/paper/2404.10904/) (2024)
- [End-to-end Semantic-centric Video-based Multimodal Affective Computing](https://mlsanity.com/paper/2408.07694/) (2024)
- [Weakly-supervised Multi-task Learning for Multimodal Affect Recognition](https://mlsanity.com/paper/2104.11560/) (2021)
- [From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation](https://mlsanity.com/paper/2110.08270/) (2021)
- [Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention](https://mlsanity.com/paper/2111.05890/) (2021)
- [LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences](https://mlsanity.com/paper/2112.01697/) (2021)
- [Multilevel Transformer For Multimodal Emotion Recognition](https://mlsanity.com/paper/2211.07711/) (2022)
- [FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference](https://mlsanity.com/paper/2209.10170/) (2022)

## Abstract (arXiv)

Existing works on multimodal affective computing tasks, such as emotion recognition, generally adopt a two-phase pipeline, first extracting feature representations for each single modality with hand-crafted algorithms and then performing end-to-end learning with the extracted features. However, the extracted features are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extraction algorithms does not generalize or scale well to different tasks, which can lead to sub-optimal performance. In this paper, we develop a fully end-to-end model that connects the two phases and optimizes them jointly. In addition, we restructure the current datasets to enable the fully end-to-end training. Furthermore, to reduce the computational overhead brought by the end-to-end model, we introduce a sparse cross-modal attention mechanism for the feature extraction. Experimental results show that our fully end-to-end model significantly surpasses the current state-of-the-art models based on the two-phase pipeline. Moreover, by adding the sparse cross-modal attention, our model can maintain performance with around half the computation in the feature extraction part.