mlsanityData to 14 Oct 2026

Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

arXiv 2309.10294 ↗First posted 19 Sep 2023cs.CL cs.AI cs.SDcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS. First, we investigated the representation ability of different speech self-supervised pre-trained models, and we found that data2vec has a good representation ability on the SER task. Second, we employed a powerful large language model (LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate emotionally congruent text and speech. We carefully designed the text prompt and dataset construction, to obtain the synthetic emotional speech data with high quality. Third, we studied different ways of data augmentation to promote the SER task with synthetic speech, including random mixing, adversarial training, transfer learning, and curriculum learning. Experiments and ablation studies on the IEMOCAP dataset demonstrate the effectiveness of our method, compared with other data augmentation methods, and data augmentation with other synthetic data.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

34 of 43 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  12. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
22 more
  1. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  2. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  3. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  4. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  5. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 49
  6. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  7. 2020x-vectors meet emotions: A study on dependencies between emotion and speaker recognitioncited by 31
  8. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  9. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  10. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  11. 2023Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Modelscited by 16
  12. 2022Speaker Normalization for Self-supervised Speech Emotion Recognitioncited by 14
  13. 2019Direct Modelling of Speech Emotion from Raw Speechcited by 13
  14. 2022Multitask Learning from Augmented Auxiliary Data for Improving Speech Emotion Recognitioncited by 11
  15. 2020Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-corpus Setting for Speech Emotion Recognitioncited by 10
  16. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9
  17. 2018On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networkscited by 8
  18. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  19. 2023Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representationcited by 7
  20. 2023A Preliminary Study on Augmenting Speech Emotion Recognition using a Diffusion Modelcited by 6
  21. 2022TESSP: Text-Enhanced Self-Supervised Speech Pre-trainingcited by 4
  22. 2023Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversationscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 6.

12 papers.

  1. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  2. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  3. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 12
  4. 2024Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selectioncited by 10
  5. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  6. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  7. 2024Reject Threshold Adaptation for Open-Set Model Attribution of Deepfake Audiocited by 1
  8. 2025Labels Generated by Large Language Models Help Measure People's Empathy in Vitrocited by 1
  9. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  10. 2024Efficient VoIP Communications through LLM-based Real-Time Speech Reconstruction and Call Prioritization for Emergency Servicescited by 0
  11. 2025Unraveling Emotions with Pre-Trained Modelscited by 0
  12. 2025Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematicscited by 0