# From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

arXiv: https://arxiv.org/abs/2610.01864 · HTML: https://mlsanity.com/paper/2610.01864/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- Mechanistic interpretability

## Builds on (most cited first, 20 of 20)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization](https://mlsanity.com/paper/2501.01108/) (2025)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment](https://mlsanity.com/paper/2502.03714/) (2025)
- [A Foundation Model for Music Informatics](https://mlsanity.com/paper/2311.03318/) (2023)
- [MARBLE: Music Audio Representation Benchmark for Universal Evaluation](https://mlsanity.com/paper/2306.10548/) (2023)
- [CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages](https://mlsanity.com/paper/2502.10362/) (2025)
- [Do Music Generation Models Encode Music Theory?](https://mlsanity.com/paper/2410.00872/) (2024)
- [Discovering and Steering Interpretable Concepts in Large Generative Music Models](https://mlsanity.com/paper/2505.18186/) (2025)
- [STONE: Self-supervised Tonality Estimator](https://mlsanity.com/paper/2407.07408/) (2024)
- [AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders](https://mlsanity.com/paper/2602.05027/) (2026)
- [S-KEY: Self-supervised Learning of Major and Minor Keys from Audio](https://mlsanity.com/paper/2501.12907/) (2025)
- [Universal Music Representations? Evaluating Foundation Models on World Music Corpora](https://mlsanity.com/paper/2506.17055/) (2025)

## Abstract (arXiv)

How can we understand what a music foundation model has learned textitinternally? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as textitstructured relations rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, we shift from feature identification to structure-based analysis, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.