mlsanityData to 7 Oct 2026

MuLan: A Joint Embedding of Music Audio and Natural Language

arXiv 2208.12415 ↗First posted 26 Aug 2022eess.AS cs.CL cs.SDcited by 85

Not on the map. Why?

Abstract (arXiv)

Music tagging and content-based retrieval systems have traditionally been constructed using pre-defined ontologies covering a rigid set of music attributes or text queries. This paper presents MuLan: a first attempt at a new generation of acoustic models that link music audio directly to unconstrained natural language music descriptions. MuLan takes the form of a two-tower, joint audio-text embedding model trained using 44 million music recordings (370K hours) and weakly-associated, free-form text annotations. Through its compatibility with a wide range of music genres and text styles (including conventional music tags), the resulting audio-text representation subsumes existing ontologies while graduating to true zero-shot functionalities. We demonstrate the versatility of the MuLan embeddings with a range of experiments including transfer learning, zero-shot music tagging, language understanding in the music domain, and cross-modal retrieval applications.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 26 Aug 2022.

Builds on

32 of 48 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2020Jukebox: A Generative Model for Musiccited by 535
  12. 2018Universal Sentence Encodercited by 485
20 more
  1. 2019Clotho: An Audio Captioning Datasetcited by 354
  2. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  3. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  4. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  5. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  6. 2021Contrastive Learning of Musical Representationscited by 81
  7. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  8. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  9. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  10. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  11. 2020Evaluation of CNN-based Automatic Music Tagging Modelscited by 39
  12. 2017Transfer learning for music classification and regression taskscited by 36
  13. 2022Text-Driven Separation of Arbitrary Soundscited by 23
  14. 2020Zero-Shot Audio Classification via Semantic Embeddingscited by 21
  15. 2020COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representationscited by 13
  16. 2021Learning music audio representations via weak language supervisioncited by 13
  17. 2021Enriched Music Representations with Multiple Cross-modal Contrastive Learningcited by 10
  18. 2020Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tagscited by 8
  19. 2021Emotion Embedding Spaces for Matching Music to Storiescited by 8
  20. 2017Multi-label Music Genre Classification from Audio, Text, and Images Using Deep Featurescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 26; 2025: 18; 2026: 12.

85 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 480
  2. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  3. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  4. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  6. 2023Getting aligned on representational alignmentcited by 142
  7. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  8. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  9. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  10. 2023Efficient Neural Music Generationcited by 50
  11. 2023MuseCoco: Generating Symbolic Music from Textcited by 50
  12. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
38 more
  1. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  2. 2024Foundation Models for Music: A Surveycited by 34
  3. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  4. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  5. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  6. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  7. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  8. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 21
  9. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  10. 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 20
  11. 2023Content-based Controls For Music Large Language Modelingcited by 19
  12. 2024MidiCaps: A large-scale MIDI dataset with text captionscited by 19
  13. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  14. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  15. 2023Language-Guided Music Recommendation for Video via Prompt Analogiescited by 18
  16. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  17. 2024MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Musiccited by 15
  18. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 15
  19. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  20. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  21. 2023DISCO-10M: A Large-Scale Music Datasetcited by 13
  22. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  23. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 13
  24. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  25. 2023Brain2Music: Reconstructing Music from Human Brain Activitycited by 10
  26. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 10
  27. 2023Talk the Walk: Synthetic Data Generation for Conversational Music Recommendationcited by 9
  28. 2023A Survey of AI Music Generation Tools and Modelscited by 9
  29. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  30. 2024Diversity-Rewarded CFG Distillationcited by 9
  31. 2025Live Music Modelscited by 9
  32. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  33. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 7
  34. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 7
  35. 2022TimbreCLIP: Connecting Timbre to Text and Imagescited by 5
  36. 2023Joint Music and Language Attention Models for Zero-shot Music Taggingcited by 5
  37. 2023MAQA: A Multimodal QA Benchmark for Negationcited by 4
  38. 2024I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognitioncited by 4

50 most cited of 85 All 85 (JSON)