mlsanityData to 7 Oct 2026

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

arXiv 2408.16532 ↗First posted 29 Aug 2024eess.AS cs.LG cs.MMcited by 124

Not on the map. Why?

Abstract (arXiv)

Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 61 of its 85 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

61 of 85 references.

References · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
38 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  8. 2024Qwen2-Audio Technical Reportcited by 543
  9. 2020Jukebox: A Generative Model for Musiccited by 535
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2023Simple and Controllable Music Generationcited by 480
  13. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  14. 2023MusicLM: Generating Music From Textcited by 442
  15. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  16. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  17. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  18. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  19. 2022AudioGen: Textually Guided Audio Generationcited by 292
  20. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  21. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  22. 2016FMA: A Dataset For Music Analysiscited by 205
  23. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  24. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  25. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  26. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  27. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  28. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  29. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  30. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  31. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  32. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  33. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  34. 2018AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmarkcited by 72
  35. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 71
  36. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  37. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  38. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 51; 2026: 60.

124 papers.

Sort:

By area:

  1. 2025Step-Audio 2 Technical Reportcited by 101
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 77
  4. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  5. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  6. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  7. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  8. 2025VibeVoice Technical Reportcited by 28
  9. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  10. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  11. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  12. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
38 more
  1. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  2. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  3. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  4. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  5. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  6. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  7. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
  8. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  9. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  10. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  11. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 10
  12. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  13. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 8
  14. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  15. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLMcited by 7
  16. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generationcited by 7
  17. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  18. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 7
  19. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  20. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  21. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  22. 2024Semantic Residual for Multimodal Unified Discrete Representationcited by 5
  23. 2025Spoken Language Modeling with Duration-Penalized Self-Supervised Unitscited by 5
  24. 2025SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codeccited by 5
  25. 2025Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoderscited by 5
  26. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancementcited by 5
  27. 2026UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Modelscited by 5
  28. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  29. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 5
  30. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  31. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  32. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  33. 2025Token-Based Audio Inpainting via Discrete Diffusioncited by 3
  34. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  35. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignmentcited by 3
  36. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  37. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 3
  38. 2026dots.tts Technical Reportcited by 3

50 most cited of 124 All 124 (JSON)