mlsanityData to 14 Oct 2026

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

arXiv 2303.17395 ↗First posted 30 Mar 2023eess.AS cs.CL cs.MMcited by 267

Not on the map. Why?

Cited by 43 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https://github.com/XinhaoMei/WavCaps.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). Builds on 63 of its 102 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

63 of 102 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2021AST: Audio Spectrogram Transformercited by 643
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  13. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2023MusicLM: Generating Music From Textcited by 448
  16. 2018nocaps: novel object captioning at scalecited by 444
  17. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  18. 2019Clotho: An Audio Captioning Datasetcited by 358
  19. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  20. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  21. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  22. 2022AudioGen: Textually Guided Audio Generationcited by 295
  23. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  24. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  25. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  26. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  27. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
  28. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 85
  29. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  30. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  31. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  32. 2021Audio Captioning Transformercited by 55
  33. 2021Sound Event Detection: A Tutorialcited by 51
  34. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  35. 2019Temporal Reasoning via Audio Question Answeringcited by 39
  36. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  37. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
  38. 2014Acoustic Scene Classificationcited by 26

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 73; 2025: 91; 2026: 78.

267 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  7. 2025Kimi-Audio Technical Reportcited by 222
  8. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  9. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  10. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  11. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  12. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
38 more
  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  3. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  4. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  5. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  6. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  7. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  8. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 88
  9. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  10. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  11. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  12. 2023Audio-Visual LLM for Video Understandingcited by 77
  13. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  14. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  15. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  16. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  17. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  18. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  19. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  20. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  21. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  22. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  23. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  24. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 46
  25. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 46
  26. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  27. 2023Separate Anything You Describecited by 43
  28. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  29. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  30. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  31. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  32. 2023AudioSR: Versatile Audio Super-resolution at Scalecited by 39
  33. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  34. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  35. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 34
  36. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 33
  37. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 31
  38. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29

50 most cited of 267 All 267 (JSON)