mlsanityData to 14 Oct 2026

MelNet: A Generative Model for Audio in the Frequency Domain

arXiv 1906.01083 ↗First posted 4 Jun 2019eess.AS cs.LG cs.SDcited by 42

Not on the map. Why?

Abstract (arXiv)

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 20; 2022: 10; 2023: 4; 2024: 1; 2025: 5; 2026: 1.

42 papers.

Sort:

By area:

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  2. 2021A Survey on Neural Speech Synthesiscited by 184
  3. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  4. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 73
  5. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  6. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  7. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesiscited by 37
  8. 2021Controllable deep melody generation via hierarchical music structure representationcited by 30
  9. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  10. 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
  11. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  12. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
30 more
  1. 2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
  2. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  3. 2022Testing Human Ability To Detect Deepfake Images of Human Facescited by 15
  4. 2022Audio representations for deep learning in sound synthesis: A reviewcited by 9
  5. 2022GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioningcited by 9
  6. 2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
  7. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  8. 2021Learnable MFCCs for Speaker Verificationcited by 6
  9. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  10. 2021How Deep Are the Fakes? Focusing on Audio Deepfake: A Surveycited by 6
  11. 2023Msanii: High Fidelity Music Synthesis on a Shoestring Budgetcited by 6
  12. 2021DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANscited by 5
  13. 2021MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GANcited by 4
  14. 2021CycleDRUMS: Automatic Drum Arrangement For Bass Lines Using CycleGANcited by 4
  15. 2022Dual Learning Music Composition and Dance Choreographycited by 3
  16. 2022A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectivescited by 3
  17. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  18. 2022cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrogramscited by 2
  19. 2023AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videoscited by 2
  20. 2021Musical Speech: A Transformer-based Composition Toolcited by 1
  21. 2025High-Fidelity Synthetic ECG Generation via Mel-Spectrogram Informed Diffusion Trainingcited by 1
  22. 2026Optimal-Transport-Guided Functional Flow Matching for Turbulent Field Generation in Hilbert Spacecited by 1
  23. 2021Improving the expressiveness of neural vocoding with non-affine Normalizing Flowscited by 0
  24. 2021Multi-Scale Spectrogram Modelling for Neural Text-to-Speechcited by 0
  25. 2021Deep Generative Models for Vehicle Speed Trajectoriescited by 0
  26. 2022R-MelNet: Reduced Mel-Spectral Modeling for Neural TTScited by 0
  27. 2022Pathway to Future Symbiotic Creativitycited by 0
  28. 2025LoopGen: Training-Free Loopable Music Generationcited by 0
  29. 2025MARS: Sound Generation via Multi-Channel Autoregression on Spectrogramscited by 0
  30. 2025Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Spacecited by 0