mlsanityData to 14 Oct 2026

A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion

arXiv 2307.11584 ↗First posted 21 Jul 2023cs.SD cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Speech Emotion Recognition (SER) is a challenging task. In this paper, we introduce a modality conversion concept aimed at enhancing emotion recognition performance on the MELD dataset. We assess our approach through two experiments: first, a method named Modality-Conversion that employs automatic speech recognition (ASR) systems, followed by a text classifier; second, we assume perfect ASR output and investigate the impact of modality conversion on SER, this method is called Modality-Conversion++. Our findings indicate that the first method yields substantial results, while the second method outperforms state-of-the-art (SOTA) speech-based approaches in terms of SER weighted-F1 (WF1) score on the MELD dataset. This research highlights the potential of modality conversion for tasks that can be conducted in alternative modalities.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

15 of 18 references.

References · 15

  1. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  2. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  3. 2018Multimodal Speech Emotion Recognition Using Audio and Textcited by 55
  4. 2022M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversationcited by 47
  5. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  6. 2021CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversationcited by 35
  7. 2022Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversationcited by 31
  8. 2022SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speechcited by 20
  9. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  10. 2023DST: Deformable Speech Transformer for Emotion Recognitioncited by 11
  11. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 9
  12. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
3 more
  1. 2023DWFormer: Dynamic Window transFormer for Speech Emotion Recognitioncited by 7
  2. 2019Multimodal Speech Emotion Recognition and Ambiguity Resolutioncited by 4
  3. 2022Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledgecited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2025RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samplescited by 7
  2. 2025Killing it with Zero-Shot: Adversarially Robust Novelty Detectioncited by 4
  3. 2023Imaginations of WALL-E : Reconstructing Experiences with an Imagination-Inspired Module for Advanced AI Systemscited by 2
  4. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  5. 2025On the Contribution of Lexical Features to Speech Emotion Recognitioncited by 0