mlsanityData to 14 Oct 2026

Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

arXiv 2211.06687 ↗First posted 12 Nov 2022cs.SD eess.AScited by 719

Not on the map. Why?

Cited by 117 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.

Key facts. Cited by 719 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Nov 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 74; 2024: 182; 2025: 216; 2026: 245.

719 papers.

Sort:

By area:

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  2. 2023Simple and Controllable Music Generationcited by 488
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  6. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  7. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  10. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  11. 2024Stable Audio Opencited by 197
  12. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
38 more
  1. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  2. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  3. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  4. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  5. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  6. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  7. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  8. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  9. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  10. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  11. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  12. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  13. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  14. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  15. 2024Long-form music generation with latent diffusioncited by 89
  16. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  17. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 80
  18. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 78
  19. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  20. 2023Audio-Visual LLM for Video Understandingcited by 77
  21. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  22. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  23. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  24. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  25. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  26. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  27. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
  28. 2024Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversioncited by 60
  29. 2022MAViL: Masked Audio-Video Learnerscited by 59
  30. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 56
  31. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  32. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  33. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  34. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  35. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  36. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  37. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 46
  38. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45

50 most cited of 719 All 719 (JSON)