mlsanityData to 7 Oct 2026

Clotho: An Audio Captioning Dataset

arXiv 1910.09387 ↗First posted 21 Oct 2019cs.SD cs.CL cs.LGcited by 354

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (https://zenodo.org/record/3490684).

Key facts. Cited by 354 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 31; 2023: 62; 2024: 66; 2025: 100; 2026: 82.

354 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2024Qwen2-Audio Technical Reportcited by 543
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2022AudioGen: Textually Guided Audio Generationcited by 292
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
38 more
  1. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  2. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  3. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  4. 2025Kimi-Audio Technical Reportcited by 210
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  7. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  8. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  9. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  10. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  11. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  12. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  13. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  14. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  15. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  16. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  17. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  18. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  19. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  20. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  21. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  22. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  23. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  24. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  25. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  26. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  27. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  28. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  29. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  30. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  31. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  32. 2023Audio-Visual LLM for Video Understandingcited by 76
  33. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  34. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  35. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  36. 2022MAViL: Masked Audio-Video Learnerscited by 56
  37. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  38. 2021Audio Captioning Transformercited by 53

50 most cited of 354 All 354 (JSON)