# Audio Retrieval with Natural Language Queries: A Benchmark Study

arXiv: https://arxiv.org/abs/2112.09418 · HTML: https://mlsanity.com/paper/2112.09418/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2021.

## Built on by (most cited first, 30 of 61)

- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images](https://mlsanity.com/paper/2306.07257/) (2023)
- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [RECAP: Retrieval-Augmented Audio Captioning](https://mlsanity.com/paper/2309.09836/) (2023)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval](https://mlsanity.com/paper/2401.13478/) (2024)
- [MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration](https://mlsanity.com/paper/2204.08058/) (2022)
- [Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks](https://mlsanity.com/paper/2310.11612/) (2023)
- [SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description](https://mlsanity.com/paper/2408.13608/) (2024)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [Data-Efficient Multimodal Fusion on a Single GPU](https://mlsanity.com/paper/2312.10144/) (2023)
- [BLAB: Brutally Long Audio Bench](https://mlsanity.com/paper/2505.03054/) (2025)
- [Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation](https://mlsanity.com/paper/2405.10084/) (2024)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization](https://mlsanity.com/paper/2508.12149/) (2025)

## Abstract (arXiv)

The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a given written description and vice versa. Text-audio retrieval enables users to search large databases through an intuitive interface: they simply issue free-form natural language descriptions of the sound they would like to hear. To study the tasks of text-audio and audio-text retrieval, which have received limited attention in the existing literature, we introduce three challenging new benchmarks. We first construct text-audio and audio-text retrieval benchmarks from the AudioCaps and Clotho audio captioning datasets. Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho. We employ these three benchmarks to establish baselines for cross-modal text-audio and audio-text retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into audio retrieval with free-form text queries. Code, audio features for all datasets used, and the SoundDescs dataset are publicly available at https://github.com/akoepke/audio-retrieval-benchmark.