Prevailing Research Areas for Music AI in the Era of Foundation Models
Not on the map. Why?
Abstract (arXiv)
Parallel to rapid advancements in foundation model research, the past few years have witnessed a surge in music AI applications. As AI-generated and AI-augmented music become increasingly mainstream, many researchers in the music AI community may wonder: what research frontiers remain unexplored? This paper outlines several key areas within music AI research that present significant opportunities for further investigation. We begin by examining foundational representation models and highlight emerging efforts toward explainability and interpretability. We then discuss the evolution toward multimodal systems, provide an overview of the current landscape of music datasets and their limitations, and address the growing importance of model efficiency in both training and deployment. Next, we explore applied directions, focusing first on generative models. We review recent systems, their computational constraints, and persistent challenges related to evaluation and controllability. We then examine extensions of these generative approaches to multimodal settings and their integration into artists' workflows, including applications in music editing, captioning, production, transcription, source separation, performance, discovery, and education. Finally, we explore copyright implications of generative music and propose strategies to safeguard artist rights. While not exhaustive, this survey aims to illuminate promising research directions enabled by recent developments in music foundation models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 89 of its 153 references that are in the map. First posted on arXiv 14 Sep 2024.
Builds on
89 of 153 references.
References · 89
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2017Neural Discrete Representation Learningcited by 4,650
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
38 more
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2020Jukebox: A Generative Model for Musiccited by 535
- 2023Simple and Controllable Music Generationcited by 480
- 2023MusicLM: Generating Music From Textcited by 442
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
- 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
- 2016FMA: A Dataset For Music Analysiscited by 205
- 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 200
- 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
- 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
- 2022Hybrid Transformers for Music Source Separationcited by 134
- 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
- 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
- 2022A Survey on the Fairness of Recommender Systemscited by 102
- 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
- 2019Music Source Separation in the Waveform Domaincited by 98
- 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
- 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
- 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
- 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
- 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
- 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 77
- 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 72
- 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 72
- 2022Music Source Separation with Band-split RNNcited by 69
- 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
- 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
- 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
- 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
- 2024Symbolic Music Generation with Non-Differentiable Rule Guided Diffusioncited by 52
- 2023MuseCoco: Generating Symbolic Music from Textcited by 50
- 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
- 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
- 2023A Foundation Model for Music Informaticscited by 45
- 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
- 2023WavMark: Watermarking for Audio Generationcited by 41
50 most cited of 89 All 89 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.