# TADA! Tuning Audio Diffusion Models through Activation Steering

arXiv: https://arxiv.org/abs/2602.11910 · HTML: https://mlsanity.com/paper/2602.11910/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 63 of its 74 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)
- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models](https://mlsanity.com/paper/2311.12092/) (2023)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)

## Built on by (most cited first, 4 of 4)

- [A Quantized Native Runtime for On-Device Semantic Audio Generation](https://mlsanity.com/paper/2607.08526/) (2026)
- [Controlling Speaking Rate in Autoregressive TTS via Activation Steering](https://mlsanity.com/paper/2609.33810/) (2026)
- [Ghost in the Encoder: Decodable Artist Identity Representations in Lyrics-to-Song Generation](https://mlsanity.com/paper/2609.39552/) (2026)
- [Harmonic Representations in Symbolic Music Transformers are Localized and Controllable](https://mlsanity.com/paper/2610.15957/) (2026)

## Abstract (arXiv)

Audio diffusion models can synthesize high-fidelity music from text, yet achieving fine-grained control over specific musical attributes remains challenging, as their internal mechanisms for representing high-level concepts are poorly understood. In this work, we use activation patching to demonstrate that recent audio diffusion architectures exhibit a semantic bottleneck, where a small, shared subset of consecutive attention layers controls distinct musical concepts, such as the presence of specific instruments, vocals, or genres. Building on this, we systematically evaluate a broad spectrum of steering paradigms, comparing activation steering against prompt-level, score-space, and weight-space interventions, analyzing the interaction between the steering mechanism and the intervention site. Our new benchmark, supported by an extensive user study, demonstrates that localized activation steering establishes a new state-of-the-art in audio concept modulation.