# SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval

arXiv: https://arxiv.org/abs/2609.02343 · HTML: https://mlsanity.com/paper/2609.02343/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [The Benefit Of Temporally-Strong Labels In Audio Event Classification](https://mlsanity.com/paper/2105.07031/) (2021)
- [AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models](https://mlsanity.com/paper/2304.00830/) (2023)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Models](https://mlsanity.com/paper/2411.18953/) (2024)
- [Improving Text-To-Audio Models with Synthetic Captions](https://mlsanity.com/paper/2406.15487/) (2024)
- [Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions](https://mlsanity.com/paper/2407.04416/) (2024)
- [Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning](https://mlsanity.com/paper/2205.05357/) (2022)
- [FoleyBench: A Benchmark For Video-to-Audio Models](https://mlsanity.com/paper/2511.13219/) (2025)
- [Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets](https://mlsanity.com/paper/2308.04258/) (2023)
- [SAO-Instruct: Free-form Audio Editing using Natural Language Instructions](https://mlsanity.com/paper/2510.22795/) (2025)
- [Woosh: A Sound Effects Foundation Model](https://mlsanity.com/paper/2604.01929/) (2026)
- [Audio-Language Datasets of Scenes and Events: A Survey](https://mlsanity.com/paper/2407.06947/) (2024)
- [SALT: Standardized Audio event Label Taxonomy](https://mlsanity.com/paper/2409.11746/) (2024)
- [Audiocards: Structured Metadata Improves Audio Language Models For Sound Design](https://mlsanity.com/paper/2602.13835/) (2026)
- [In-Context Prompt Editing For Conditional Audio Generation](https://mlsanity.com/paper/2311.00895/) (2023)
- [Aligning Audio Captions with Human Preferences](https://mlsanity.com/paper/2509.14659/) (2025)

## Abstract (arXiv)

Recent advances in audio-language modeling have been driven by large-scale audio captioning datasets. However, existing datasets remain limited by low semantic diversity, generic descriptions lacking acoustic details, and one-to-one audio-caption mappings that poorly reflect the inherent ambiguity of auditory perception. We introduce SonicCaps, a large-scale audio captioning dataset comprising ~15M captions paired with ~700k audio clips, generated using a multi-modal large language model (Qwen3-Omni) conditioned on both audio and text. To explicitly promote diversity, we generate around 24 captions per audio via structured prompt engineering and few- shot generation, spanning main descriptions, rephrased variants (verbosity, style) and semantic tags. Human evaluation shows that SonicCaps is rated significantly higher than existing captioning datasets, with fine-grained analyses indicating that our captions are perceived as more descriptive and precise, which strongly correlates with quality judgments. Finally, training CLAP models on SonicCaps with a multi-caption sampling strategy consistently improves audio retrieval and zero-shot classification, with stronger generalization across public and commercial benchmarks. We release both SonicCaps and two specialized CLAP models on hugging face: https://huggingface.co/datasets/Zineb/SonicCaps.