# From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion

arXiv: https://arxiv.org/abs/2308.02560 · HTML: https://mlsanity.com/paper/2308.02560/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)

## Built on by (most cited first, 22 of 22)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Boosting Latent Diffusion with Perceptual Objectives](https://mlsanity.com/paper/2411.04873/) (2024)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Audio Super-Resolution with Latent Bridge Models](https://mlsanity.com/paper/2509.17609/) (2025)
- [PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation](https://mlsanity.com/paper/2408.07547/) (2024)
- [Generative De-Quantization for Neural Speech Codec via Latent Diffusion](https://mlsanity.com/paper/2311.08330/) (2023)
- [RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction](https://mlsanity.com/paper/2403.05010/) (2024)
- [Learning Source Disentanglement in Neural Audio Codec](https://mlsanity.com/paper/2409.11228/) (2024)
- [Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization](https://mlsanity.com/paper/2408.08019/) (2024)
- [Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation](https://mlsanity.com/paper/2512.23278/) (2025)
- [PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders](https://mlsanity.com/paper/2404.02702/) (2024)
- [FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates](https://mlsanity.com/paper/2409.17635/) (2024)
- [Generative AI Training and Copyright Law](https://mlsanity.com/paper/2502.15858/) (2025)
- [A Language Modeling Approach to Diacritic-Free Hebrew TTS](https://mlsanity.com/paper/2407.12206/) (2024)
- [Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model](https://mlsanity.com/paper/2511.20470/) (2025)
- [Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study](https://mlsanity.com/paper/2406.04633/) (2024)
- [On the Design of Diffusion-based Neural Speech Codecs](https://mlsanity.com/paper/2504.08470/) (2025)
- [Compression of Higher Order Ambisonics with Multichannel RVQGAN](https://mlsanity.com/paper/2411.12008/) (2024)
- [Toward Agentic Environments: GenAI and the Convergence of AI, Sustainability, and Human-Centric Spaces](https://mlsanity.com/paper/2512.15787/) (2025)
- [Latent Fourier Transform](https://mlsanity.com/paper/2604.17986/) (2026)
- [Geometric Iterative Retrieval for Neural Audio Codec Resynthesis](https://mlsanity.com/paper/2608.19141/) (2026)

## Abstract (arXiv)

Deep generative models can generate high-fidelity audio conditioned on various types of representations (e.g., mel-spectrograms, Mel-frequency Cepstral Coefficients (MFCC)). Recently, such models have been used to synthesize audio waveforms conditioned on highly compressed representations. Although such methods produce impressive results, they are prone to generate audible artifacts when the conditioning is flawed or imperfect. An alternative modeling approach is to use diffusion models. However, these have mainly been used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating relatively low sampling rate signals. In this work, we propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality (e.g., speech, music, environmental sounds) from low-bitrate discrete representations. At equal bit rate, the proposed approach outperforms state-of-the-art generative techniques in terms of perceptual quality. Training and, evaluation code, along with audio samples, are available on the facebookresearch/audiocraft Github page.