mlsanityData to 19 Oct 2026

AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook

arXiv 2509.21968 ↗First posted 26 Sep 2025eess.AS cs.SDcited by 4

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

We propose AUV, a unified neural audio codec with a single codebook, which enables a favourable reconstruction of speech and further extends to general audio, including vocal, music, and sound. AUV is capable of tackling any 16 kHz mixed-domain audio segment at bit rates around 700 bps. To accomplish this, we guide the matryoshka codebook with nested domain-specific partitions, assigned with corresponding teacher models to perform distillation, all in a single-stage training. A conformer-style encoder-decoder architecture with STFT features as audio representation is employed, yielding better audio quality. Comprehensive evaluations demonstrate that AUV exhibits comparable audio reconstruction ability to state-of-the-art domain-specific single-layer quantizer codecs, showcasing the potential of audio universal vector quantization with a single codebook. The pre-trained model and demo samples are available at https://swivid.github.io/AUV/.

Key facts. Cited by 4 papers in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 45 of its 49 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

45 of 49 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation31 May
  2. 2025DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation19 May
  3. 2025UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook27 Feb
  4. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis6 Feb
  5. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Coding29 Nov
  6. 2024TS3-Codec: Transformer-Based Simple Streaming Single Codec27 Nov
  7. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  8. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec9 Sep
  9. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model30 Aug
  10. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling29 Aug
  11. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation11 Jun
  12. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models31 Aug
4 more
  1. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec26 May
  2. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision7 Feb
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  4. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 29

  1. 2017Neural Discrete Representation Learningcited by 4,843
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,282
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  8. 2022High Fidelity Neural Audio Compressioncited by 917
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 736
  11. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  12. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
17 more
  1. 2025Qwen3-Omni Technical Reportcited by 578
  2. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 422
  4. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 412
  5. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  6. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 218
  7. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 162
  8. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 155
  9. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 84
  10. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 73
  11. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 58
  12. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 44
  13. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 37
  14. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 21
  15. 2025FlowDec: A flow-based full-band general audio codec with high perceptual qualitycited by 21
  16. 2024MuCodec: Ultra Low-Bitrate Music Codeccited by 11
  17. 2025Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Modelscited by 5

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantizationcited by 13
  2. 2026OmniCodec: Low Frame Rate Universal Audio Codec with Semantic-Acoustic Disentanglementcited by 2
  3. 2026LLM-Codec: Neural Audio Codec Meets Language Model Objectivescited by 0
  4. 2026ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs
  2. 2026UniStream: Multi-Expert Residual Vector Quantization for 48 kHz Causal Streaming Audio Coding
  3. 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates
  4. 2025QINCODEC: Neural Audio Compression with Implicit Neural Codebooks
  5. 2024NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization
  6. 2025MelTok: 2D Tokenization for Single-Codebook Audio Compression
  7. 2025Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machine
  8. 2025DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners