# Latent Space Disentanglement via Activation Steering for Interpretable Attribute Control in Symbolic Music Generation

arXiv: https://arxiv.org/abs/2605.31295 · HTML: https://mlsanity.com/paper/2605.31295/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Artificial Intelligence in the Creative Industries: A Review](https://mlsanity.com/paper/2007.12391/) (2020)
- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation](https://mlsanity.com/paper/2406.10970/) (2024)
- [Seed-Music: A Unified Framework for High Quality and Controlled Music Generation](https://mlsanity.com/paper/2409.09214/) (2024)
- [XMusic: Towards a Generalized and Controllable Symbolic Music Generation Framework](https://mlsanity.com/paper/2501.08809/) (2025)
- [A database linking piano and orchestral MIDI scores with application to automatic projective orchestration](https://mlsanity.com/paper/1810.08611/) (2018)
- [SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers](https://mlsanity.com/paper/2404.02252/) (2024)
- [Activation Patching for Interpretable Steering in Music Generation](https://mlsanity.com/paper/2504.04479/) (2025)
- [Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time](https://mlsanity.com/paper/2512.04748/) (2025)
- [Steering Autoregressive Music Generation with Recursive Feature Machines](https://mlsanity.com/paper/2510.19127/) (2025)
- [Composer Vector: Style-steering Symbolic Music Generation in a Latent Space](https://mlsanity.com/paper/2604.03333/) (2026)
- [How Do Large Language Models Learn Concepts During Continual Pre-Training?](https://mlsanity.com/paper/2601.03570/) (2026)

## Built on by (most cited first, 1 of 1)

- [MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering](https://mlsanity.com/paper/2608.06638/) (2026)

## Abstract (arXiv)

Transformer-based architectures have significantly advanced the generation of complex symbolic sequences, yet a significant gap remains in achieving fine-grained, interpretable control over discrete signal attributes. This paper investigates the mechanistic interpretability of the Multitrack Music Transformer (MMT) and proposes a framework for deterministic attribute modulation without retraining to bridge this gap via inference-time activation steering. Utilizing the Difference-in-Means (DiffMean) methodology, we isolate latent directions for signal attributes, specifically Pitch and Duration, within the residual stream. We validate the Linear Representation Hypothesis in this domain, achieving high correlation between steering magnitude and attribute shift. To address the inherent feature entanglement in multi-attribute steering, we introduce a Dual Steering framework utilizing Gram-Schmidt Orthogonalization. Experimental results demonstrate that this geometric decoupling reduces conceptual interference and signal degradation compared to naive vector addition, enabling independent deterministic control even against strong autoregressive conditioning.