# RAVE: A variational autoencoder for fast and high-quality neural audio synthesis

arXiv: https://arxiv.org/abs/2111.05011 · HTML: https://mlsanity.com/paper/2111.05011/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://mlsanity.com/paper/2005.05106/) (2020)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [MelNet: A Generative Model for Audio in the Frequency Domain](https://mlsanity.com/paper/1906.01083/) (2019)
- [DurIAN: Duration Informed Attention Network For Multimodal Synthesis](https://mlsanity.com/paper/1909.01700/) (2019)
- [SING: Symbol-to-Instrument Neural Generator](https://mlsanity.com/paper/1810.09785/) (2018)
- [A Universal Music Translation Network](https://mlsanity.com/paper/1805.07848/) (2018)
- [Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics](https://mlsanity.com/paper/1805.08501/) (2018)

## Built on by (most cited first, 30 of 73)

- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [SingSong: Generating musical accompaniments from singing](https://mlsanity.com/paper/2301.12662/) (2023)
- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [Musika! Fast Infinite Waveform Music Generation](https://mlsanity.com/paper/2208.08706/) (2022)
- [MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models](https://mlsanity.com/paper/2406.04673/) (2024)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Long-Term Rhythmic Video Soundtracker](https://mlsanity.com/paper/2305.01319/) (2023)
- [OmniAudio: Generating Spatial Audio from 360-Degree Video](https://mlsanity.com/paper/2504.14906/) (2025)
- [Bass Accompaniment Generation via Latent Diffusion](https://mlsanity.com/paper/2402.01412/) (2024)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks](https://mlsanity.com/paper/2211.01839/) (2022)
- [Hypernetworks build Implicit Neural Representations of Sounds](https://mlsanity.com/paper/2302.04959/) (2023)
- [Efficient Autoregressive Audio Modeling via Next-Scale Prediction](https://mlsanity.com/paper/2408.09027/) (2024)
- [Improving Musical Accompaniment Co-creation via Diffusion Transformers](https://mlsanity.com/paper/2410.23005/) (2024)
- [Combining audio control and style transfer using latent diffusion](https://mlsanity.com/paper/2408.00196/) (2024)
- [SoK: How Robust is Audio Watermarking in Generative AI models?](https://mlsanity.com/paper/2503.19176/) (2025)
- [Live Music Models](https://mlsanity.com/paper/2508.04651/) (2025)
- [Streamable Neural Audio Synthesis With Non-Causal Convolutions](https://mlsanity.com/paper/2204.07064/) (2022)
- [DiffMoog: a Differentiable Modular Synthesizer for Sound Matching](https://mlsanity.com/paper/2401.12570/) (2024)
- [MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AI](https://mlsanity.com/paper/2507.03599/) (2025)
- [Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders](https://mlsanity.com/paper/2510.23802/) (2025)
- [GOAT: A Large Dataset of Paired Guitar Audio Recordings and Tablatures](https://mlsanity.com/paper/2509.22655/) (2025)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Learning to Upsample and Upmix Audio in the Latent Domain](https://mlsanity.com/paper/2506.00681/) (2025)
- [DrumGAN VST: A Plugin for Drum Sound Analysis/Synthesis With Autoencoding Generative Adversarial Networks](https://mlsanity.com/paper/2206.14723/) (2022)

## Abstract (arXiv)

Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks. However, as raw waveform modelling remains an inherently difficult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU. We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.