# DExter: Learning and Controlling Performance Expression with Diffusion Models arXiv: https://arxiv.org/abs/2406.14850 ยท HTML: https://mlsanity.com/paper/2406.14850/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Audio, speech and music, one of 14 directions in Diffusion models. - Builds on 24 of its 43 references that are in the map. - First posted on arXiv 21 Jun 2024. ## Placed in - Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/) ## Builds on (most cited first, 24 of 24) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017) - [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020) - [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020) - [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023) - [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022) - [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021) - [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018) - [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021) - [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022) - [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023) - [Partitura: A Python Package for Symbolic Music Processing](https://mlsanity.com/paper/2206.01071/) (2022) - [Discrete Diffusion Probabilistic Models for Symbolic Music Generation](https://mlsanity.com/paper/2305.09489/) (2023) - [Symbolic Music Representations for Classification Tasks: A Systematic Evaluation](https://mlsanity.com/paper/2309.02567/) (2023) - [DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability](https://mlsanity.com/paper/2210.05148/) (2022) - [Towards Explainable Music Emotion Recognition: The Route via Mid-level Features](https://mlsanity.com/paper/1907.03572/) (2019) - [A data-driven approach to mid-level perceptual musical feature modeling](https://mlsanity.com/paper/1806.04903/) (2018) - [On Perceived Emotion in Expressive Piano Performance: Further Experimental Evidence for the Relevance of Mid-level Perceptual Features](https://mlsanity.com/paper/2107.13231/) (2021) - [Sketching the Expression: Flexible Rendering of Expressive Piano Performance with Self-Supervised Learning](https://mlsanity.com/paper/2208.14867/) (2022) - [The ACCompanion: Combining Reactivity, Robustness, and Musical Expressivity in an Automatic Piano Accompanist](https://mlsanity.com/paper/2304.12939/) (2023) - [Sounding Out Reconstruction Error-Based Evaluation of Generative Models of Expressive Performance](https://mlsanity.com/paper/2401.00471/) (2023) ## Built on by (most cited first, 11 of 11) - [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024) - [MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling](https://mlsanity.com/paper/2507.08530/) (2025) - [AudioMorphix: Training-free audio editing with diffusion probabilistic models](https://mlsanity.com/paper/2505.16076/) (2025) - [Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings](https://mlsanity.com/paper/2409.07827/) (2024) - [SyMuPe: Affective and Controllable Symbolic Music Performance](https://mlsanity.com/paper/2511.03425/) (2025) - [PianoCoRe: Combined and Refined Piano MIDI Dataset](https://mlsanity.com/paper/2605.06627/) (2026) - [Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder](https://mlsanity.com/paper/2507.01582/) (2025) - [EmoHeal: An End-to-End System for Personalized Therapeutic Music Retrieval from Fine-grained Emotions](https://mlsanity.com/paper/2509.15986/) (2025) - [Disentangling Score Content and Performance Style for Joint Piano Rendering and Transcription](https://mlsanity.com/paper/2509.23878/) (2025) - [Integrating Contextual Embeddings into Evaluation of Expressive MIDI Piano Performances](https://mlsanity.com/paper/2607.27909/) (2026) - [Learning to Predict Performance-induced Emotion Differences in Classical Piano Music](https://mlsanity.com/paper/2607.28876/) (2026) ## Abstract (arXiv) In the pursuit of developing expressive music performance models using artificial intelligence, this paper introduces DExter, a new approach leveraging diffusion probabilistic models to render Western classical piano performances. In this approach, performance parameters are represented in a continuous expression space and a diffusion model is trained to predict these continuous parameters while being conditioned on the musical score. Furthermore, DExter also enables the generation of interpretations (expressive variations of a performance) guided by perceptually meaningful features by conditioning jointly on score and perceptual feature representations. Consequently, we find that our model is useful for learning expressive performance, generating perceptually steered performances, and transferring performance styles. We assess the model through quantitative and qualitative analyses, focusing on specific performance metrics regarding dimensions like asynchrony and articulation, as well as through listening tests comparing generated performances with different human interpretations. Results show that DExter is able to capture the time-varying correlation of the expressive parameters, and compares well to existing rendering models in subjectively evaluated ratings. The perceptual-feature-conditioned generation and transferring capabilities of DExter are verified by a proxy model predicting perceptual characteristics of differently steered performances.