# Emotion recognition by fusing time synchronous and time asynchronous representations

arXiv: https://arxiv.org/abs/2010.14102 · HTML: https://mlsanity.com/paper/2010.14102/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 26 of 26)

- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information](https://mlsanity.com/paper/2203.15326/) (2022)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques](https://mlsanity.com/paper/2406.08353/) (2024)
- [Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models](https://mlsanity.com/paper/2405.10025/) (2024)
- [Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation](https://mlsanity.com/paper/2108.02510/) (2021)
- [Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression](https://mlsanity.com/paper/2306.06760/) (2023)
- [Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.13661/) (2023)
- [Affect Recognition in Conversations Using Large Language Models](https://mlsanity.com/paper/2309.12881/) (2023)
- [Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors](https://mlsanity.com/paper/2203.04443/) (2022)
- [Climate and Weather: Inspecting Depression Detection via Emotion Recognition](https://mlsanity.com/paper/2204.14099/) (2022)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attention](https://mlsanity.com/paper/2404.13509/) (2024)
- [Learning Robust Self-attention Features for Speech Emotion Recognition with Label-adaptive Mixup](https://mlsanity.com/paper/2305.06273/) (2023)
- [Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation](https://mlsanity.com/paper/2402.12862/) (2024)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation](https://mlsanity.com/paper/2402.11747/) (2024)
- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)
- [A Persian ASR-based SER: Modification of Sharif Emotional Speech Database and Investigation of Persian Text Corpora](https://mlsanity.com/paper/2211.09956/) (2022)
- [MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition](https://mlsanity.com/paper/2306.09361/) (2023)
- [Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weighting](https://mlsanity.com/paper/2411.17674/) (2024)
- [WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2412.05558/) (2024)

## Abstract (arXiv)

In this paper, a novel two-branch neural network model structure is proposed for multimodal emotion recognition, which consists of a time synchronous branch (TSB) and a time asynchronous branch (TAB). To capture correlations between each word and its acoustic realisation, the TSB combines speech and text modalities at each input window frame and then does pooling across time to form a single embedding vector. The TAB, by contrast, provides cross-utterance information by integrating sentence text embeddings from a number of context utterances into another embedding vector. The final emotion classification uses both the TSB and the TAB embeddings. Experimental results on the IEMOCAP dataset demonstrate that the two-branch structure achieves state-of-the-art results in 4-way classification with all common test setups. When using automatic speech recognition (ASR) output instead of manually transcribed reference text, it is shown that the cross-utterance information considerably improves the robustness against ASR errors. Furthermore, by incorporating an extra class for all the other emotions, the final 5-way classification system with ASR hypotheses can be viewed as a prototype for more realistic emotion recognition systems.