# Prompting Audios Using Acoustic Properties For Emotion Representation

arXiv: https://arxiv.org/abs/2310.02298 · HTML: https://mlsanity.com/paper/2310.02298/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 8 of 8)

- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [What is the ground truth? Reliability of multi-annotator data for audio tagging](https://mlsanity.com/paper/2104.04214/) (2021)
- [The phonetic bases of vocal expressed emotion: natural versus acted](https://mlsanity.com/paper/1911.05733/) (2019)
- [Detecting gender differences in perception of emotion in crowdsourced data](https://mlsanity.com/paper/1910.11386/) (2019)

## Built on by (most cited first, 3 of 3)

- [Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models](https://mlsanity.com/paper/2609.00727/) (2026)
- [EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast](https://mlsanity.com/paper/2505.23732/) (2025)
- [Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style](https://mlsanity.com/paper/2508.11187/) (2025)

## Abstract (arXiv)

Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.