# Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2201.11826 · HTML: https://mlsanity.com/paper/2201.11826/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 11 of 11)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [x-vectors meet emotions: A study on dependencies between emotion and speaker recognition](https://mlsanity.com/paper/2002.05039/) (2020)
- [Contrastive Unsupervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2102.06357/) (2021)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotions](https://mlsanity.com/paper/2002.01323/) (2020)
- [Sentiment Analysis on Speaker Specific Speech Data](https://mlsanity.com/paper/1802.06209/) (2018)
- [Reusing Neural Speech Representations for Auditory Emotion Recognition](https://mlsanity.com/paper/1803.11508/) (2018)

## Built on by (most cited first, 8 of 8)

- [Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression](https://mlsanity.com/paper/2306.06760/) (2023)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation](https://mlsanity.com/paper/2207.03334/) (2022)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition](https://mlsanity.com/paper/2310.00283/) (2023)
- [GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2405.02151/) (2024)
- [Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions](https://mlsanity.com/paper/2503.22711/) (2025)
- [Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition](https://mlsanity.com/paper/2405.00307/) (2024)

## Abstract (arXiv)

We propose a novel multi-task pre-training method for Speech Emotion Recognition (SER). We pre-train SER model simultaneously on Automatic Speech Recognition (ASR) and sentiment classification tasks to make the acoustic ASR model more ``emotion aware''. We generate targets for the sentiment classification using text-to-sentiment model trained on publicly available data. Finally, we fine-tune the acoustic ASR on emotion annotated speech data. We evaluated the proposed approach on the MSP-Podcast dataset, where we achieved the best reported concordance correlation coefficient (CCC) of 0.41 for valence prediction.