# Audio Retrieval with WavText5K and CLAP Training

arXiv: https://arxiv.org/abs/2209.14275 · HTML: https://mlsanity.com/paper/2209.14275/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 28 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [What is the ground truth? Reliability of multi-annotator data for audio tagging](https://mlsanity.com/paper/2104.04214/) (2021)
- [Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection](https://mlsanity.com/paper/2008.07085/) (2020)
- [Improving weakly supervised sound event detection with self-supervised auxiliary tasks](https://mlsanity.com/paper/2106.06858/) (2021)

## Built on by (most cited first, 30 of 35)

- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [AudioLCM: Text-to-Audio Generation with Latent Consistency Models](https://mlsanity.com/paper/2406.00356/) (2024)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping](https://mlsanity.com/paper/2309.10667/) (2023)
- [PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2408.09481/) (2024)
- [M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2409.05929/) (2024)
- [Data-Efficient Multimodal Fusion on a Single GPU](https://mlsanity.com/paper/2312.10144/) (2023)
- [Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation](https://mlsanity.com/paper/2405.10084/) (2024)
- [Bridging Language Gaps in Audio-Text Retrieval](https://mlsanity.com/paper/2406.07012/) (2024)
- [IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling](https://mlsanity.com/paper/2506.00736/) (2025)
- [Describing emotions with acoustic property prompts for speech emotion recognition](https://mlsanity.com/paper/2211.07737/) (2022)
- [FLAP: Fast Language-Audio Pre-training](https://mlsanity.com/paper/2311.01615/) (2023)
- [Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction](https://mlsanity.com/paper/2507.09834/) (2025)
- [Prompting Audios Using Acoustic Properties For Emotion Representation](https://mlsanity.com/paper/2310.02298/) (2023)
- [SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization](https://mlsanity.com/paper/2509.21033/) (2025)
- [Efficient and High-Fidelity Omni Modality Retrieval](https://mlsanity.com/paper/2603.02098/) (2026)
- [CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition](https://mlsanity.com/paper/2310.11830/) (2023)
- [Generalized zero-shot audio-to-intent classification](https://mlsanity.com/paper/2311.02482/) (2023)

## Abstract (arXiv)

Audio-Text retrieval takes a natural language query to retrieve relevant audio files in a database. Conversely, Text-Audio retrieval takes an audio file as a query to retrieve relevant natural language descriptions. Most of the literature train retrieval systems with one audio captioning dataset, but evaluating the benefit of training with multiple datasets is underexplored. Moreover, retrieval systems have to learn the alignment between elaborated sentences describing audio content of variable length ranging from a few seconds to several minutes. In this work, we propose a new collection of web audio-text pairs and a new framework for retrieval. First, we provide a new collection of about five thousand web audio-text pairs that we refer to as WavText5K. When used to train our retrieval system, WavText5K improved performance more than other audio captioning datasets. Second, our framework learns to connect language and audio content by using a text encoder, two audio encoders, and a contrastive learning objective. Combining both audio encoders helps to process variable length audio. The two contributions beat state of the art performance for AudioCaps and Clotho on Text-Audio retrieval by a relative 2% and 16%, and Audio-Text retrieval by 6% and 23%.