# GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2306.07848 · HTML: https://mlsanity.com/paper/2306.07848/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition](https://mlsanity.com/paper/2211.08233/) (2022)
- [Light-SERNet: A lightweight fully convolutional neural network for speech emotion recognition](https://mlsanity.com/paper/2110.03435/) (2021)
- [Speaker Normalization for Self-supervised Speech Emotion Recognition](https://mlsanity.com/paper/2202.01252/) (2022)
- [Key-Sparse Transformer for Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2106.11532/) (2021)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.13661/) (2023)
- [Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.09856/) (2023)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)

## Built on by (most cited first, 10 of 10)

- [Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models](https://mlsanity.com/paper/2409.12139/) (2024)
- [ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimization](https://mlsanity.com/paper/2506.01032/) (2025)
- [Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matching](https://mlsanity.com/paper/2411.02026/) (2024)
- [GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.20919/) (2025)
- [ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech](https://mlsanity.com/paper/2505.13805/) (2025)
- [Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling](https://mlsanity.com/paper/2410.01350/) (2024)
- [Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusion](https://mlsanity.com/paper/2504.05164/) (2025)
- [EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast](https://mlsanity.com/paper/2505.23732/) (2025)
- [Can We Estimate Purchase Intention Based on Zero-shot Speech Emotion Recognition?](https://mlsanity.com/paper/2410.09636/) (2024)
- [Audio emotion recognition for atypical hearing](https://mlsanity.com/paper/2609.31168/) (2026)

## Abstract (arXiv)

Contrastive cross-modality pretraining has recently exhibited impressive success in diverse fields, whereas there is limited research on their merits in speech emotion recognition (SER). In this paper, we propose GEmo-CLAP, a kind of gender-attribute-enhanced contrastive language-audio pretraining (CLAP) method for SER. Specifically, we first construct an effective emotion CLAP (Emo-CLAP) for SER, using pre-trained text and audio encoders. Second, given the significance of gender information in SER, two novel multi-task learning based GEmo-CLAP (ML-GEmo-CLAP) and soft label based GEmo-CLAP (SL-GEmo-CLAP) models are further proposed to incorporate gender information of speech signals, forming more reasonable objectives. Experiments on IEMOCAP indicate that our proposed two GEmo-CLAPs consistently outperform Emo-CLAP with different pre-trained models. Remarkably, the proposed WavLM-based SL-GEmo-CLAP obtains the best WAR of 83.16%, which performs better than state-of-the-art SER methods.