mlsanityData to 7 Oct 2026

The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation

arXiv 2311.10057 ↗First posted 16 Nov 2023cs.SD cs.AI cs.CLcited by 68

Not on the map. Why?

Abstract (arXiv)

We introduce the Song Describer dataset (SDD), a new crowdsourced corpus of high-quality audio-caption pairs, designed for the evaluation of music-and-language models. The dataset consists of 1.1k human-written natural language descriptions of 706 music recordings, all publicly accessible and released under Creative Common licenses. To showcase the use of our dataset, we benchmark popular models on three key music-and-language tasks (music captioning, text-to-music generation and music-language retrieval). Our experiments highlight the importance of cross-dataset evaluation and offer insights into how researchers can use SDD to gain a broader understanding of model performance.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

21 of 36 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  7. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  8. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  9. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  10. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  11. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  12. 2017DeLiGAN : Generative Adversarial Networks for Diverse and Limited Datacited by 77
9 more
  1. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  2. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  3. 2022Contrastive Audio-Language Learning for Musiccited by 32
  4. 2023Audio-Text Models Do Not Yet Leverage Natural Languagecited by 26
  5. 2021MusCaps: Generating Captions for Music Audiocited by 23
  6. 2022Toward Universal Text-to-Music Retrievalcited by 18
  7. 2023Language-Guided Music Recommendation for Video via Prompt Analogiescited by 18
  8. 2021Learning music audio representations via weak language supervisioncited by 13
  9. 2023Data leakage in cross-modal retrieval training: A case studycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 15; 2026: 30.

68 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2024Stable Audio Opencited by 192
  3. 2024Long-form music generation with latent diffusioncited by 88
  4. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  5. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  6. 2024Foundation Models for Music: A Surveycited by 34
  7. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  8. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  9. 2024SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformerscited by 19
  10. 2024FLUX that Plays Musiccited by 18
  11. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  12. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
38 more
  1. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  2. 2025JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadatacited by 14
  3. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  4. 2025Audio Super-Resolution with Latent Bridge Modelscited by 13
  5. 2024CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Modelscited by 12
  6. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  7. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  8. 2024Diversity-Rewarded CFG Distillationcited by 9
  9. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  10. 2025Live Music Modelscited by 9
  11. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  12. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generationcited by 6
  13. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  14. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  15. 2024Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentationcited by 5
  16. 2026Stable Audio 3cited by 5
  17. 2024Evaluation of pretrained language models on music understandingcited by 4
  18. 2024CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentationcited by 4
  19. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  20. 2025Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoderscited by 4
  21. 2025Steering Autoregressive Music Generation with Recursive Feature Machinescited by 4
  22. 2024Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learningcited by 3
  23. 2024Sound Check: Auditing Audio Datasetscited by 3
  24. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  25. 2026AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Followingcited by 3
  26. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 3
  27. 2025Exploring State-Space-Model based Language Model in Music Generationcited by 2
  28. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  29. 2026SAME: A Semantically-Aligned Music Autoencodercited by 2
  30. 2024MetaBGM: Dynamic Soundtrack Transformation For Continuous Multi-Scene Experiences With Ambient Awareness And Personalizationcited by 1
  31. 2026UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representationcited by 1
  32. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 1
  33. 2026HumMusQA: A Human-written Music Understanding QA Benchmark Datasetcited by 1
  34. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generatorscited by 1
  35. 2025LenslessMic: Audio Encryption and Authentication via Lensless Computational Imagingcited by 0
  36. 2026ConceptCaps: a Distilled Concept Dataset for Interpretability in Music Modelscited by 0
  37. 2026Training-Efficient Text-to-Music Generation with State-Space Modelingcited by 0
  38. 2026IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrievalcited by 0

50 most cited of 68 All 68 (JSON)