# BAMU: Bitstream-Aware Marginal-Utility Allocation for Frozen Pretrained Neural Speech Codecs

arXiv: https://arxiv.org/abs/2608.08432 · HTML: https://mlsanity.com/paper/2608.08432/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 11 of 11)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric](https://mlsanity.com/paper/2004.09584/) (2020)
- [AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec](https://mlsanity.com/paper/2305.16608/) (2023)
- [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://mlsanity.com/paper/2309.07405/) (2023)
- [FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates](https://mlsanity.com/paper/2510.00981/) (2025)
- [Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate](https://mlsanity.com/paper/2505.16845/) (2025)
- [Variable Bitrate Residual Vector Quantization for Audio Coding](https://mlsanity.com/paper/2410.06016/) (2024)
- [Improving Test-Time Performance of RVQ-based Neural Codecs](https://mlsanity.com/paper/2509.19186/) (2025)
- [SPG-Codec: Exploring the Role and Boundaries of Semantic Priors in Ultra-Low-Bitrate Neural Speech Coding](https://mlsanity.com/paper/2604.26296/) (2026)

## Built on by (most cited first, 1 of 1)

- [UniStream: Multi-Expert Residual Vector Quantization for 48 kHz Causal Streaming Audio Coding](https://mlsanity.com/paper/2609.09866/) (2026)

## Abstract (arXiv)

Pretrained neural speech codecs typically use a fixed residual vector quantization (RVQ) depth for all frames, ignoring temporal variation in quantization difficulty. We propose BAMU, a bitstream-aware dynamic RVQ allocation framework for frozen pretrained codecs. A lightweight, rate-independent predictor estimates frame- and layer-wise marginal latent-distortion reductions, while a constrained allocator selects prefix-valid depths under an exact serialized-size budget. Experiments on EnCodec and DAC over LibriSpeech, together with VCTK evaluation, show consistent EnCodec gains and DAC improvements mainly at medium and high rates. A 30-listener study confirms a MOS improvement from 3.449 to 3.780 over matched fixed-depth coding.