mlsanityData to 14 Oct 2026

A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives

arXiv 2504.00837 ↗First posted 1 Apr 2025cs.SD cs.AI cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

Multi-modal music generation, using multiple modalities like text, images, and video alongside musical scores and audio as guidance, is an emerging research area with broad applications. This paper reviews this field, categorizing music generation systems from the perspective of modalities. The review covers modality representation, multi-modal data alignment, and their utilization to guide music generation. Current datasets and evaluation methods are also discussed. Key challenges in this area include effective multi-modal integration, large-scale comprehensive datasets, and systematic evaluation methods. Finally, an outlook on future research directions is provided, focusing on creativity, efficiency, multi-modal alignment, and evaluation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 154 of its 221 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

154 of 221 references.

References · 154

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  13. 2020Extracting Training Data from Large Language Modelscited by 2,189
  14. 2014Conditional Generative Adversarial Netscited by 2,162
  15. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  16. 2018SlowFast Networks for Video Recognitioncited by 1,867
  17. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  18. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  19. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  20. 2021ViViT: A Video Vision Transformercited by 1,620
  21. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  22. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  23. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  24. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  25. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  26. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  27. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  28. 2018Self-Attention with Relative Position Representationscited by 1,120
  29. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  30. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  31. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  32. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  33. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  34. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  35. 2022High Fidelity Neural Audio Compressioncited by 687
  36. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  37. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  38. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602

50 most cited of 154 All 154 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generationcited by 3
  2. 2025Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approachcited by 0
  3. 2026LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generationcited by 0
  4. 2026MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generationcited by 0