mlsanityData to 7 Oct 2026

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

arXiv 2408.09438 ↗First posted 18 Aug 2024cs.MM cs.AI cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

21 of 32 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  12. 2020Multi-modal Attention for Speech Emotion Recognitioncited by 29
9 more
  1. 2022COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognitioncited by 22
  2. 2015EmoNets: Multimodal deep learning approaches for emotion recognition in videocited by 20
  3. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  4. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 8
  5. 2022MMER: Multimodal Multi-task Learning for Speech Emotion Recognitioncited by 7
  6. 2021Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognitioncited by 6
  7. 2023Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysiscited by 5
  8. 2023Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognitioncited by 4
  9. 2022Multilevel Transformer For Multimodal Emotion Recognitioncited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraintscited by 1
  2. 2025DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysiscited by 1