# Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling

arXiv: https://arxiv.org/abs/2007.15474 · HTML: https://mlsanity.com/paper/2007.15474/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 30 of 45)

- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model](https://mlsanity.com/paper/2311.00968/) (2023)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control](https://mlsanity.com/paper/2202.05528/) (2022)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [Symbolic music generation conditioned on continuous-valued emotions](https://mlsanity.com/paper/2203.16165/) (2022)
- [Generating music with sentiment using Transformer-GANs](https://mlsanity.com/paper/2212.11134/) (2022)
- [GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers](https://mlsanity.com/paper/2302.05393/) (2023)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Exploring XAI for the Arts: Explaining Latent Space in Generative Music](https://mlsanity.com/paper/2308.05496/) (2023)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [Conditional Drums Generation using Compound Word Representations](https://mlsanity.com/paper/2202.04464/) (2022)
- [Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges](https://mlsanity.com/paper/2406.08809/) (2024)
- [Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses](https://mlsanity.com/paper/2202.10453/) (2022)
- [Learning long-term music representations via hierarchical contextual constraints](https://mlsanity.com/paper/2202.06180/) (2022)
- [EmoGen: Eliminating Subjective Bias in Emotional Music Generation](https://mlsanity.com/paper/2307.01229/) (2023)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [ProgGP: From GuitarPro Tablature Neural Generation To Progressive Metal Production](https://mlsanity.com/paper/2307.05328/) (2023)
- [Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI](https://mlsanity.com/paper/2311.08336/) (2023)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [Fast Diffusion GAN Model for Symbolic Music Generation Controlled by Emotions](https://mlsanity.com/paper/2310.14040/) (2023)
- [Between the AI and Me: Analysing Listeners' Perspectives on AI- and Human-Composed Progressive Metal Music](https://mlsanity.com/paper/2407.21615/) (2024)

## Abstract (arXiv)

High-level musical qualities (such as emotion) are often abstract, subjective, and hard to quantify. Given these difficulties, it is not easy to learn good feature representations with supervised learning techniques, either because of the insufficiency of labels, or the subjectiveness (and hence large variance) in human-annotated labels. In this paper, we present a framework that can learn high-level feature representations with a limited amount of data, by first modelling their corresponding quantifiable low-level attributes. We refer to our proposed framework as Music FaderNets, which is inspired by the fact that low-level attributes can be continuously manipulated by separate "sliding faders" through feature disentanglement and latent regularization techniques. High-level features are then inferred from the low-level representations through semi-supervised clustering using Gaussian Mixture Variational Autoencoders (GM-VAEs). Using arousal as an example of a high-level feature, we show that the "faders" of our model are disentangled and change linearly w.r.t. the modelled low-level attributes of the generated output music. Furthermore, we demonstrate that the model successfully learns the intrinsic relationship between arousal and its corresponding low-level attributes (rhythm and note density), with only 1% of the training set being labelled. Finally, using the learnt high-level feature representations, we explore the application of our framework in style transfer tasks across different arousal states. The effectiveness of this approach is verified through a subjective listening test.