# Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI

arXiv: https://arxiv.org/abs/2311.08336 · HTML: https://mlsanity.com/paper/2311.08336/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [A Functional Taxonomy of Music Generation Systems](https://mlsanity.com/paper/1812.04186/) (2018)
- [Deep Music Analogy Via Latent Representation Disentanglement](https://mlsanity.com/paper/1906.03626/) (2019)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)
- [Music Generation by Deep Learning - Challenges and Directions](https://mlsanity.com/paper/1712.04371/) (2017)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017)
- [To Explain or Not to Explain: A Study on the Necessity of Explanations for Autonomous Vehicles](https://mlsanity.com/paper/2006.11684/) (2020)
- [ExplAIn: Explanatory Artificial Intelligence for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2008.05731/) (2020)
- [Look Who's Talking Now: Implications of AV's Explanations on Driver's Trust, AV Preference, Anxiety and Mental Workload](https://mlsanity.com/paper/1905.08878/) (2019)
- [In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound](https://mlsanity.com/paper/1902.06442/) (2019)
- [Exploring XAI for the Arts: Explaining Latent Space in Generative Music](https://mlsanity.com/paper/2308.05496/) (2023)
- [Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions](https://mlsanity.com/paper/2002.02393/) (2020)
- [Algorithmic Composition of Melodies with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1606.07251/) (2016)
- [Getting Closer to the Essence of Music: The Con Espressione Manifesto](https://mlsanity.com/paper/1611.09733/) (2016)
- [Rethinking Recurrent Latent Variable Model for Music Composition](https://mlsanity.com/paper/1810.03226/) (2018)
- [Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders](https://mlsanity.com/paper/2106.08867/) (2021)
- [Jacobian Norm with Selective Input Gradient Regularization for Improved and Interpretable Adversarial Defense](https://mlsanity.com/paper/2207.13036/) (2022)

## Built on by (most cited first, 4 of 4)

- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Exploring the Needs of Practising Musicians in Co-Creative AI Through Co-Design](https://mlsanity.com/paper/2502.09055/) (2025)
- [Reducing Barriers to the Use of Marginalised Music Genres in AI](https://mlsanity.com/paper/2407.13439/) (2024)
- [CoLyricist: Enhancing Lyric Writing with AI through Workflow-Aligned Support](https://mlsanity.com/paper/2602.22606/) (2026)

## Abstract (arXiv)

Generative AI models for music and the arts in general are increasingly complex and hard to understand. The field of eXplainable AI (XAI) seeks to make complex and opaque AI models such as neural networks more understandable to people. One approach to making generative AI models more understandable is to impose a small number of semantically meaningful attributes on generative AI models. This paper contributes a systematic examination of the impact that different combinations of Variational Auto-Encoder models (MeasureVAE and AdversarialVAE), configurations of latent space in the AI model (from 4 to 256 latent dimensions), and training datasets (Irish folk, Turkish folk, Classical, and pop) have on music generation performance when 2 or 4 meaningful musical attributes are imposed on the generative model. To date there have been no systematic comparisons of such models at this level of combinatorial detail. Our findings show that MeasureVAE has better reconstruction performance than AdversarialVAE which has better musical attribute independence. Results demonstrate that MeasureVAE was able to generate music across music genres with interpretable musical dimensions of control, and performs best with low complexity music such a pop and rock. We recommend that a 32 or 64 latent dimensional space is optimal for 4 regularised dimensions when using MeasureVAE to generate music across genres. Our results are the first detailed comparisons of configurations of state-of-the-art generative AI models for music and can be used to help select and configure AI models, musical features, and datasets for more understandable generation of music.