# ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

arXiv: https://arxiv.org/abs/2409.09213 · HTML: https://mlsanity.com/paper/2409.09213/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model](https://mlsanity.com/paper/2304.13731/) (2023)
- [Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification](https://mlsanity.com/paper/1608.04363/) (2016)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [CochlScene: Acquisition of acoustic scene data using crowdsourcing](https://mlsanity.com/paper/2211.02289/) (2022)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [RECAP: Retrieval-Augmented Audio Captioning](https://mlsanity.com/paper/2309.09836/) (2023)
- [On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning](https://mlsanity.com/paper/2408.13068/) (2024)

## Built on by (most cited first, 12 of 12)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors](https://mlsanity.com/paper/2502.14627/) (2025)
- [PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs](https://mlsanity.com/paper/2506.10423/) (2025)
- [SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization](https://mlsanity.com/paper/2509.21033/) (2025)
- [Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap](https://mlsanity.com/paper/2510.11330/) (2025)
- [EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast](https://mlsanity.com/paper/2505.23732/) (2025)
- [WavLink: Compact Audio-Text Embeddings with a Global Whisper Token](https://mlsanity.com/paper/2601.15118/) (2026)
- [TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control](https://mlsanity.com/paper/2603.09332/) (2026)
- [Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition](https://mlsanity.com/paper/2603.23057/) (2026)
- [Drift-Augmented Scoring: Text-Derived Noise Robustness for Zero-Shot Audio-Language Classification](https://mlsanity.com/paper/2606.04844/) (2026)
- [Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift](https://mlsanity.com/paper/2608.15037/) (2026)

## Abstract (arXiv)

Open-vocabulary audio-language models, like CLAP, offer a promising approach for zero-shot audio classification (ZSAC) by enabling classification with any arbitrary set of categories specified with natural language prompts. In this paper, we propose a simple but effective method to improve ZSAC with CLAP. Specifically, we shift from the conventional method of using prompts with abstract category labels (e.g., Sound of an organ) to prompts that describe sounds using their inherent descriptive features in a diverse context (e.g.,The organ's deep and resonant tones filled the cathedral.). To achieve this, we first propose ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild. These rewritten captions describe each sound event in the original caption using their unique discriminative characteristics. ReCLAP outperforms all baselines on both multi-modal audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification with ReCLAP, we propose prompt augmentation. In contrast to the traditional method of employing hand-written template prompts, we generate custom prompts for each unique label in the dataset. These custom prompts first describe the sound event in the label and then employ them in diverse scenes. Our proposed method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all baselines by 1% - 55%.