# Music Generation by Deep Learning - Challenges and Directions

arXiv: https://arxiv.org/abs/1712.04371 · HTML: https://mlsanity.com/paper/1712.04371/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2017.

## Built on by (most cited first, 24 of 24)

- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [A Survey of AI Music Generation Tools and Models](https://mlsanity.com/paper/2308.12982/) (2023)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [The GigaMIDI Dataset with Features for Expressive Music Performance Detection](https://mlsanity.com/paper/2502.17726/) (2025)
- [Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI](https://mlsanity.com/paper/2311.08336/) (2023)
- [MusFlow: Multimodal Music Generation via Conditional Flow Matching](https://mlsanity.com/paper/2504.13535/) (2025)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [Deep Oscillatory Neural Network](https://mlsanity.com/paper/2405.03725/) (2024)
- [JAZZVAR: A Dataset of Variations found within Solo Piano Performances of Jazz Standards for Music Overpainting](https://mlsanity.com/paper/2307.09670/) (2023)
- [Automatic Generation of Product Concepts from Positive Examples, with an Application to Music Streaming](https://mlsanity.com/paper/2210.01515/) (2022)
- [AffectMachine-Classical: A novel system for generating affective classical music](https://mlsanity.com/paper/2304.04915/) (2023)
- [Exploring Latent Spaces of Tonal Music using Variational Autoencoders](https://mlsanity.com/paper/2311.03621/) (2023)
- [CoComposer: LLM Multi-agent Collaborative Music Composition](https://mlsanity.com/paper/2509.00132/) (2025)
- [MusicSwarm: Biologically Inspired Intelligence for Music Composition](https://mlsanity.com/paper/2509.11973/) (2025)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [A Novel Bi-LSTM And Transformer Architecture For Generating Tabla Music](https://mlsanity.com/paper/2404.05765/) (2024)
- [Unrolled Creative Adversarial Network For Generating Novel Musical Pieces](https://mlsanity.com/paper/2501.00452/) (2024)
- [Prime Convolutional Model: Breaking the Ground for Theoretical Explainability](https://mlsanity.com/paper/2503.02773/) (2025)
- [Parameter-Minimal Neural DE Solvers via Horner Polynomials](https://mlsanity.com/paper/2602.14737/) (2026)
- [A novel LSTM music generator based on the fractional time-frequency feature extraction](https://mlsanity.com/paper/2604.17823/) (2026)
- [RPPNet: Perceptually-Grouped Rhythm-Pitch Primitives for Long-Term Structure Melody Generation via Boundary-Aware Modeling](https://mlsanity.com/paper/2607.19776/) (2026)

## Abstract (arXiv)

In addition to traditional tasks such as prediction, classification and translation, deep learning is receiving growing attention as an approach for music generation, as witnessed by recent research groups such as Magenta at Google and CTRL (Creator Technology Research Lab) at Spotify. The motivation is in using the capacity of deep learning architectures and training techniques to automatically learn musical styles from arbitrary musical corpora and then to generate samples from the estimated distribution. However, a direct application of deep learning to generate content rapidly reaches limits as the generated content tends to mimic the training set without exhibiting true creativity. Moreover, deep learning architectures do not offer direct ways for controlling generation (e.g., imposing some tonality or other arbitrary constraints). Furthermore, deep learning architectures alone are autistic automata which generate music autonomously without human user interaction, far from the objective of interactively assisting musicians to compose and refine music. Issues such as: control, structure, creativity and interactivity are the focus of our analysis. In this paper, we select some limitations of a direct application of deep learning to music generation, analyze why the issues are not fulfilled and how to address them by possible approaches. Various examples of recent systems are cited as examples of promising directions.