# Flat Latent Manifolds for Human-machine Co-creation of Music

arXiv: https://arxiv.org/abs/2202.12243 · HTML: https://mlsanity.com/paper/2202.12243/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer](https://mlsanity.com/paper/1807.07543/) (2018)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [Learning Stochastic Recurrent Networks](https://mlsanity.com/paper/1411.7610/) (2014)
- [Metrics for Deep Generative Models](https://mlsanity.com/paper/1711.01204/) (2017)
- [Learning Hierarchical Priors in VAEs](https://mlsanity.com/paper/1905.04982/) (2019)
- [Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models](https://mlsanity.com/paper/1711.05772/) (2017)
- [Learning to Groove with Inverse Sequence Transformations](https://mlsanity.com/paper/1905.06118/) (2019)
- [On Implicit Regularization in β-VAEs](https://mlsanity.com/paper/2002.00041/) (2020)
- [Learning Flat Latent Manifolds with VAEs](https://mlsanity.com/paper/2002.04881/) (2020)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)
- [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017)
- [Fast Approximate Geodesics for Deep Generative Models](https://mlsanity.com/paper/1812.08284/) (2018)
- [Active Learning based on Data Uncertainty and Model Sensitivity](https://mlsanity.com/paper/1808.02026/) (2018)
- [Metric Learning for Temporal Sequence Alignment](https://mlsanity.com/paper/1409.3136/) (2014)
- [Inspecting and Interacting with Meaningful Music Representations using VAE](https://mlsanity.com/paper/1904.08842/) (2019)

## Built on by (most cited first, 1 of 1)

- [Two Sonification Methods for the MindCube](https://mlsanity.com/paper/2506.18196/) (2025)

## Abstract (arXiv)

The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine--musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.