# Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders

arXiv: https://arxiv.org/abs/2507.07867 · HTML: https://mlsanity.com/paper/2507.07867/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [Masked Autoencoders Are Effective Tokenizers for Diffusion Models](https://mlsanity.com/paper/2502.03444/) (2025)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling](https://mlsanity.com/paper/2502.09509/) (2025)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [VampNet: Music Generation via Masked Acoustic Token Modeling](https://mlsanity.com/paper/2307.04686/) (2023)
- [Codec-SUPERB: An In-Depth Analysis of Sound Codec Models](https://mlsanity.com/paper/2402.13071/) (2024)
- [Generation or Replication: Auscultating Audio Latent Diffusion Models](https://mlsanity.com/paper/2310.10604/) (2023)
- [Learning to Upsample and Upmix Audio in the Latent Domain](https://mlsanity.com/paper/2506.00681/) (2025)
- [An Independence-promoting Loss for Music Generation with Language Models](https://mlsanity.com/paper/2406.02315/) (2024)
- [Speech Separation using Neural Audio Codecs with Embedding Loss](https://mlsanity.com/paper/2411.17998/) (2024)

## Built on by (most cited first, 4 of 4)

- [Rethinking Music Captioning with Music Metadata LLMs](https://mlsanity.com/paper/2602.03023/) (2026)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [Learning Linearity in Audio Consistency Autoencoders via Implicit Regularization](https://mlsanity.com/paper/2510.23530/) (2025)
- [Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding](https://mlsanity.com/paper/2606.27320/) (2026)

## Abstract (arXiv)

Neural audio codecs and autoencoders have emerged as versatile models for audio compression, transmission, feature-extraction, and latent-space generation. However, a key limitation is that most are trained to maximize reconstruction fidelity, often neglecting the specific latent structure necessary for optimal performance in diverse downstream applications. We propose a simple, post-hoc framework to address this by modifying the bottleneck of a pre-trained autoencoder. Our method introduces a "Re-Bottleneck", an inner bottleneck trained exclusively through latent space losses to instill user-defined structure. We demonstrate the framework's effectiveness in three experiments. First, we enforce an ordering on latent channels without sacrificing reconstruction quality. Second, we align latents with semantic embeddings, analyzing the impact on downstream diffusion modeling. Third, we introduce equivariance, ensuring that a filtering operation on the input waveform directly corresponds to a specific transformation in the latent space. Ultimately, our Re-Bottleneck framework offers a flexible and efficient way to tailor representations of neural audio models, enabling them to seamlessly meet the varied demands of different applications with minimal additional training.