# EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast

arXiv: https://arxiv.org/abs/2505.23732 · HTML: https://mlsanity.com/paper/2505.23732/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Rank-N-Contrast: Learning Continuous Representations for Regression](https://mlsanity.com/paper/2210.01189/) (2022)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [SECap: Speech Emotion Captioning with Large Language Model](https://mlsanity.com/paper/2312.10381/) (2023)
- [Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning](https://mlsanity.com/paper/2412.07909/) (2024)
- [Training Audio Captioning Models without Audio](https://mlsanity.com/paper/2309.07372/) (2023)
- [ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds](https://mlsanity.com/paper/2409.09213/) (2024)
- [GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition](https://mlsanity.com/paper/2306.07848/) (2023)
- [ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks](https://mlsanity.com/paper/2406.07203/) (2024)
- [LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2501.11468/) (2025)
- [AlignCap: Aligning Speech Emotion Captioning to Human Preferences](https://mlsanity.com/paper/2410.19134/) (2024)
- [Prompting Audios Using Acoustic Properties For Emotion Representation](https://mlsanity.com/paper/2310.02298/) (2023)
- [Emotion-Aligned Contrastive Learning Between Images and Music](https://mlsanity.com/paper/2308.12610/) (2023)

## Built on by (most cited first, 1 of 1)

- [NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion](https://mlsanity.com/paper/2511.00256/) (2025)

## Abstract (arXiv)

Current emotion-based contrastive language-audio pretraining (CLAP) methods typically learn by na"ively aligning audio samples with corresponding text prompts. Consequently, this approach fails to capture the ordinal nature of emotions, hindering inter-emotion understanding and often resulting in a wide modality gap between the audio and text embeddings due to insufficient alignment. To handle these drawbacks, we introduce EmotionRankCLAP, a supervised contrastive learning approach that uses dimensional attributes of emotional speech and natural language prompts to jointly capture fine-grained emotion variations and improve cross-modal alignment. Our approach utilizes a Rank-N-Contrast objective to learn ordered relationships by contrasting samples based on their rankings in the valence-arousal space. EmotionRankCLAP outperforms existing emotion-CLAP methods in modeling emotion ordinality across modalities, measured via a cross-modal retrieval task.