# Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models

arXiv: https://arxiv.org/abs/1911.09762 · HTML: https://mlsanity.com/paper/1911.09762/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2019.

## Built on by (most cited first, 21 of 21)

- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue](https://mlsanity.com/paper/2312.15316/) (2023)
- [DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in Spoken Utterances](https://mlsanity.com/paper/2110.07592/) (2021)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [The Role of Phonetic Units in Speech Emotion Recognition](https://mlsanity.com/paper/2108.01132/) (2021)
- [MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attention](https://mlsanity.com/paper/2404.13509/) (2024)
- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Lego-Features: Exporting modular encoder features for streaming and deliberation ASR](https://mlsanity.com/paper/2304.00173/) (2023)
- [CTA-RNN: Channel and Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2203.17023/) (2022)
- [Spoken Language Understanding on Unseen Tasks With In-Context Learning](https://mlsanity.com/paper/2505.07731/) (2025)
- [Beyond Isolated Utterances: Conversational Emotion Recognition](https://mlsanity.com/paper/2109.06112/) (2021)
- [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025)
- [Dialog speech sentiment classification for imbalanced datasets](https://mlsanity.com/paper/2109.07228/) (2021)
- [e-Inu: Simulating A Quadruped Robot With Emotional Sentience](https://mlsanity.com/paper/2301.00964/) (2023)
- ["We care": Improving Code Mixed Speech Emotion Recognition in Customer-Care Conversations](https://mlsanity.com/paper/2308.03150/) (2023)
- [UniSLU: Unified Spoken Language Understanding from Heterogeneous Cross-Task Datasets](https://mlsanity.com/paper/2507.12951/) (2025)
- [Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization](https://mlsanity.com/paper/2507.19356/) (2025)
- [Speech Emotion Recognition via Entropy-Aware Score Selection](https://mlsanity.com/paper/2508.20796/) (2025)

## Abstract (arXiv)

In this paper, we propose to use pre-trained features from end-to-end ASR models to solve speech sentiment analysis as a down-stream task. We show that end-to-end ASR features, which integrate both acoustic and text information from speech, achieve promising results. We use RNN with self-attention as the sentiment classifier, which also provides an easy visualization through attention weights to help interpret model predictions. We use well benchmarked IEMOCAP dataset and a new large-scale speech sentiment dataset SWBD-sentiment for evaluation. Our approach improves the-state-of-the-art accuracy on IEMOCAP from 66.6% to 71.7%, and achieves an accuracy of 70.10% on SWBD-sentiment with more than 49,500 utterances.