# Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition

arXiv: https://arxiv.org/abs/2201.05834 · HTML: https://mlsanity.com/paper/2201.05834/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 15 Jan 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [SGM: Sequence Generation Model for Multi-label Classification](https://mlsanity.com/paper/1806.04822/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [Self-Supervised Model Adaptation for Multimodal Semantic Segmentation](https://mlsanity.com/paper/1808.03833/) (2018)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle](https://mlsanity.com/paper/2003.06692/) (2020)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Multi-Label Learning with Global and Local Label Correlation](https://mlsanity.com/paper/1704.01415/) (2017)
- [Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network](https://mlsanity.com/paper/1911.09075/) (2019)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [Collaboration based Multi-Label Learning](https://mlsanity.com/paper/1902.03047/) (2019)

## Built on by (most cited first, 22 of 22)

- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition](https://mlsanity.com/paper/2312.10201/) (2023)
- [Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations](https://mlsanity.com/paper/2407.04955/) (2024)
- [MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields](https://mlsanity.com/paper/2302.02978/) (2023)
- [Triple Disentangled Representation Learning for Multimodal Affective Analysis](https://mlsanity.com/paper/2401.16119/) (2024)
- [Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition](https://mlsanity.com/paper/2407.07026/) (2024)
- [RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems](https://mlsanity.com/paper/2503.04252/) (2025)
- [Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion](https://mlsanity.com/paper/2503.23721/) (2025)
- [M3PT: A Multi-Modal Model for POI Tagging](https://mlsanity.com/paper/2306.10079/) (2023)
- [Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2502.13954/) (2025)
- [Hi-EF: Benchmarking Emotion Forecasting in Human-interaction](https://mlsanity.com/paper/2407.16406/) (2024)
- [Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors](https://mlsanity.com/paper/2411.05879/) (2024)
- [Leveraging CLIP Encoder for Multimodal Emotion Recognition](https://mlsanity.com/paper/2506.00903/) (2025)
- [Incorporating Scene Context and Semantic Labels for Enhanced Group-level Emotion Recognition](https://mlsanity.com/paper/2509.21747/) (2025)
- [Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors](https://mlsanity.com/paper/2512.16485/) (2025)
- [Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition](https://mlsanity.com/paper/2412.13541/) (2024)
- [RAMer: Reconstruction-based Adversarial Model for Multi-party Multi-modal Multi-label Emotion Recognition](https://mlsanity.com/paper/2502.10435/) (2025)
- [Partitioner Guided Modal Learning Framework](https://mlsanity.com/paper/2507.11661/) (2025)
- [DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition](https://mlsanity.com/paper/2508.01644/) (2025)
- [Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment](https://mlsanity.com/paper/2606.11269/) (2026)
- [Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition](https://mlsanity.com/paper/2609.16962/) (2026)

## Abstract (arXiv)

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts. Code is available at https://github.com/kniter1/TAILOR.