# Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2501.10408 · HTML: https://mlsanity.com/paper/2501.10408/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 79 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances](https://mlsanity.com/paper/2203.06935/) (2022)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark](https://mlsanity.com/paper/2406.07162/) (2024)
- [Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languages](https://mlsanity.com/paper/1812.10411/) (2018)
- [Contrastive Unsupervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2102.06357/) (2021)
- [Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition](https://mlsanity.com/paper/2211.08233/) (2022)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [Unsupervised Adversarial Domain Adaptation for Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/1907.06083/) (2019)
- [Transformer based unsupervised pre-training for acoustic representation learning](https://mlsanity.com/paper/2007.14602/) (2020)
- [A study on cross-corpus speech emotion recognition and data augmentation](https://mlsanity.com/paper/2201.03511/) (2022)
- [Automated Screening for Distress: A Perspective for the Future](https://mlsanity.com/paper/1902.09944/) (2019)
- [The SpeakIn Speaker Verification System for Far-Field Speaker Verification Challenge 2022](https://mlsanity.com/paper/2209.11625/) (2022)

## Abstract (arXiv)

Speech Emotion Recognition (SER) plays a crucial role in enhancing human-computer interaction. Cross-Linguistic SER (CLSER) has been a challenging research problem due to significant variability in linguistic and acoustic features of different languages. In this study, we propose a novel approach HuMP-CAT, which combines HuBERT, MFCC, and prosodic characteristics. These features are fused using a cross-attention transformer (CAT) mechanism during feature extraction. Transfer learning is applied to gain from a source emotional speech dataset to the target corpus for emotion recognition. We use IEMOCAP as the source dataset to train the source model and evaluate the proposed method on seven datasets in five languages (e.g., English, German, Spanish, Italian, and Chinese). We show that, by fine-tuning the source model with a small portion of speech from the target datasets, HuMP-CAT achieves an average accuracy of 78.75% across the seven datasets, with notable performance of 88.69% on EMODB (German language) and 79.48% on EMOVO (Italian language). Our extensive evaluation demonstrates that HuMP-CAT outperforms existing methods across multiple target languages.