# Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2110.15684 · HTML: https://mlsanity.com/paper/2110.15684/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [ASR-based Features for Emotion Recognition: A Transfer Learning Approach](https://mlsanity.com/paper/1805.09197/) (2018)
- [A Transfer Learning Method for Speech Emotion Recognition from Automatic Speech Recognition](https://mlsanity.com/paper/2008.02863/) (2020)

## Built on by (most cited first, 21 of 21)

- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition](https://mlsanity.com/paper/2409.09785/) (2024)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition](https://mlsanity.com/paper/2505.00059/) (2025)
- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models](https://mlsanity.com/paper/2409.16920/) (2024)
- [Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labeling](https://mlsanity.com/paper/2409.16937/) (2024)
- [A Persian ASR-based SER: Modification of Sharif Emotional Speech Database and Investigation of Persian Text Corpora](https://mlsanity.com/paper/2211.09956/) (2022)
- [Disentangling Textual and Acoustic Features of Neural Speech Representations](https://mlsanity.com/paper/2410.03037/) (2024)
- [Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search](https://mlsanity.com/paper/2211.08237/) (2022)
- [Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition](https://mlsanity.com/paper/2305.13583/) (2023)
- [STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition](https://mlsanity.com/paper/2402.01227/) (2024)
- [Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models](https://mlsanity.com/paper/2410.03312/) (2024)
- [Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs](https://mlsanity.com/paper/2410.20334/) (2024)
- [Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction](https://mlsanity.com/paper/2506.10930/) (2025)
- [A Cross-Domain Approach for Continuous Impression Recognition from Dyadic Audio-Visual-Physio Signals](https://mlsanity.com/paper/2203.13932/) (2022)
- [On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition](https://mlsanity.com/paper/2305.12540/) (2023)
- [An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks](https://mlsanity.com/paper/2406.14747/) (2024)
- [Emotion Loss Attacking: Adversarial Attack Perception for Skeleton based on Multi-dimensional Features](https://mlsanity.com/paper/2406.19815/) (2024)
- [HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling](https://mlsanity.com/paper/2509.18570/) (2025)

## Abstract (arXiv)

Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.