# Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes

arXiv: https://arxiv.org/abs/2609.04516 · HTML: https://mlsanity.com/paper/2609.04516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Music ControlNet: Multiple Time-varying Controls for Music Generation](https://mlsanity.com/paper/2311.07069/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2405.20289/) (2024)
- [Aligning Text-to-Music Evaluation with Human Preferences](https://mlsanity.com/paper/2503.16669/) (2025)
- [Do Music Generation Models Encode Music Theory?](https://mlsanity.com/paper/2410.00872/) (2024)
- [Discovering and Steering Interpretable Concepts in Large Generative Music Models](https://mlsanity.com/paper/2505.18186/) (2025)
- [Activation Patching for Interpretable Steering in Music Generation](https://mlsanity.com/paper/2504.04479/) (2025)
- [Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders](https://mlsanity.com/paper/2510.23802/) (2025)
- [Steering Autoregressive Music Generation with Recursive Feature Machines](https://mlsanity.com/paper/2510.19127/) (2025)
- [Composer Vector: Style-steering Symbolic Music Generation in a Latent Space](https://mlsanity.com/paper/2604.03333/) (2026)

## Abstract (arXiv)

Recent work on controllable music generation has focused on autoregressive models, leaving diffusion-based systems comparatively underexplored. We present a lightweight method for steering the pitch content of audio produced by Stable Audio Open, a latent diffusion model for music synthesis. A small convolutional probe containing approximately 125k parameters is trained to decode frame-level pitch-class activations from the model's variational autoencoder latent space, using paired audio and MIDI data. At inference time, the frozen probe serves as a differentiable loss function: its gradient with respect to the denoising latent is used to nudge generation toward a user-specified pitch-class sequence, requiring no retraining or architectural modification of the base model. Across 27 evaluation trials spanning 9 text prompts and 3 target melodies, probe-guided generation increases melodic coherence by 2.4x over the unguided baseline (p < 1e-5, Wilcoxon signed-rank test), demonstrating that musically meaningful structure is both recoverable and steerable in diffusion-based music latent spaces.