# MelNet: A Generative Model for Audio in the Frequency Domain

arXiv: https://arxiv.org/abs/1906.01083 · HTML: https://mlsanity.com/paper/1906.01083/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 42)

- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment](https://mlsanity.com/paper/2406.07855/) (2024)
- [CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis](https://mlsanity.com/paper/2106.07431/) (2021)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning](https://mlsanity.com/paper/2107.09998/) (2021)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency](https://mlsanity.com/paper/2111.09052/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Testing Human Ability To Detect Deepfake Images of Human Faces](https://mlsanity.com/paper/2212.05056/) (2022)
- [Audio representations for deep learning in sound synthesis: A review](https://mlsanity.com/paper/2201.02490/) (2022)
- [GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning](https://mlsanity.com/paper/2211.05385/) (2022)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching](https://mlsanity.com/paper/2502.11128/) (2025)
- [Learnable MFCCs for Speaker Verification](https://mlsanity.com/paper/2102.10322/) (2021)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [How Deep Are the Fakes? Focusing on Audio Deepfake: A Survey](https://mlsanity.com/paper/2111.14203/) (2021)
- [Msanii: High Fidelity Music Synthesis on a Shoestring Budget](https://mlsanity.com/paper/2301.06468/) (2023)
- [DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs](https://mlsanity.com/paper/2108.01216/) (2021)
- [MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN](https://mlsanity.com/paper/2101.04785/) (2021)
- [CycleDRUMS: Automatic Drum Arrangement For Bass Lines Using CycleGAN](https://mlsanity.com/paper/2104.00353/) (2021)
- [Dual Learning Music Composition and Dance Choreography](https://mlsanity.com/paper/2201.11999/) (2022)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrograms](https://mlsanity.com/paper/2205.07319/) (2022)

## Abstract (arXiv)

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.