# A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions

arXiv: https://arxiv.org/abs/2011.06801 · HTML: https://mlsanity.com/paper/2011.06801/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2020.

## Built on by (most cited first, 30 of 56)

- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation](https://mlsanity.com/paper/2206.07771/) (2022)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Quantized GAN for Complex Music Generation from Dance Videos](https://mlsanity.com/paper/2204.00604/) (2022)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation](https://mlsanity.com/paper/2502.13128/) (2025)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography](https://mlsanity.com/paper/2107.09287/) (2021)
- [MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes](https://mlsanity.com/paper/2205.09248/) (2022)
- [MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control](https://mlsanity.com/paper/2202.05528/) (2022)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [Music Composition with Deep Learning: A Review](https://mlsanity.com/paper/2108.12290/) (2021)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [Apollo: Band-sequence Modeling for High-Quality Audio Restoration](https://mlsanity.com/paper/2409.08514/) (2024)
- [Score Transformer: Generating Musical Score from Note-level Representation](https://mlsanity.com/paper/2112.00355/) (2021)
- [YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations](https://mlsanity.com/paper/2211.07131/) (2022)
- [A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music](https://mlsanity.com/paper/2308.13736/) (2023)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [An Order-Complexity Aesthetic Assessment Model for Aesthetic-aware Music Recommendation](https://mlsanity.com/paper/2402.08300/) (2024)
- [Subjective Evaluation of Deep Learning Models for Symbolic Music Composition](https://mlsanity.com/paper/2203.14641/) (2022)
- [Learning Interpretable Low-dimensional Representation via Physical Symmetry](https://mlsanity.com/paper/2302.10890/) (2023)
- [A Comprehensive Survey on Generative AI for Video-to-Music Generation](https://mlsanity.com/paper/2502.12489/) (2025)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)

## Abstract (arXiv)

The utilization of deep learning techniques in generating various contents (such as image, text, etc.) has become a trend. Especially music, the topic of this paper, has attracted widespread attention of countless researchers.The whole process of producing music can be divided into three stages, corresponding to the three levels of music generation: score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directly. Previous surveys have explored the network models employed in the field of automatic music generation. However, the development history, the model evolution, as well as the pros and cons of same music generation task have not been clearly illustrated. This paper attempts to provide an overview of various composition tasks under different music generation levels, covering most of the currently popular music generation tasks using deep learning. In addition, we summarize the datasets suitable for diverse tasks, discuss the music representations, the evaluation methods as well as the challenges under different levels, and finally point out several future directions.