mlsanityData to 7 Oct 2026

Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition

arXiv 2207.04697 ↗First posted 11 Jul 2022cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

The research and applications of multimodal emotion recognition have become increasingly popular recently. However, multimodal emotion recognition faces the challenge of lack of data. To solve this problem, we propose to use transfer learning which leverages state-of-the-art pre-trained models including wav2vec 2.0 and BERT for this task. Multi-level fusion approaches including coattention-based early fusion and late fusion with the models trained on both embeddings are explored. Also, a multi-granularity framework which extracts not only frame-level speech embeddings but also segment-level embeddings including phone, syllable and word-level speech embeddings is proposed to further boost the performance. By combining our coattention-based early fusion model and late fusion model with the multi-granularity feature extraction framework, we obtain result that outperforms best baseline approaches by 1.3% unweighted accuracy (UA) on the IEMOCAP dataset.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

18 of 36 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016Layer Normalizationcited by 5,579
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  9. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  10. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  11. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  12. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
6 more
  1. 2019Leveraging BERT for Extractive Text Summarization on Lecturescited by 63
  2. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  3. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  4. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  5. 2019Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worldscited by 19
  6. 2022Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognitioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 2; 2025: 0; 2026: 2.

9 papers.

  1. 2023LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Modelscited by 13
  2. 2023GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognitioncited by 10
  3. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  4. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 8
  5. 2022Multilevel Transformer For Multimodal Emotion Recognitioncited by 2
  6. 2024Multimodal Belief Predictioncited by 1
  7. 2024Efficient infusion of self-supervised representations in Automatic Speech Recognitioncited by 0
  8. 2026Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detectioncited by 0
  9. 2026When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependencecited by 0