mlsanityData to 7 Oct 2026

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

arXiv 1704.01279 ↗First posted 5 Apr 2017cs.LG cs.AI cs.SDcited by 222

Not on the map. Why?

Abstract (arXiv)

Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 36; 2023: 33; 2024: 42; 2025: 43; 2026: 32.

222 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2021Neural Fields in Visual Computing and Beyondcited by 539
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  5. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  6. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  7. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  8. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  9. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  10. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  11. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  12. 2021Contrastive Learning of Musical Representationscited by 81
38 more
  1. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  2. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 77
  3. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 72
  4. 2021LEAF: A Learnable Frontend for Audio Classificationcited by 69
  5. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  6. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  7. 2021Learning Signal-Agnostic Manifolds of Neural Fieldscited by 45
  8. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  9. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  10. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  11. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  12. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  13. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  14. 2022Diagnosing and Fixing Manifold Overfitting in Deep Generative Modelscited by 28
  15. 2022Learning strides in convolutional neural networkscited by 25
  16. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  17. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  18. 2022Musika! Fast Infinite Waveform Music Generationcited by 24
  19. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  20. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  21. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  22. 2023JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generationcited by 17
  23. 2025The iNaturalist Sounds Datasetcited by 17
  24. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  25. 2021Large-scale optimal transport map estimation using projection pursuitcited by 16
  26. 2022Evaluating generative audio systems and their metricscited by 16
  27. 2021Cartoon Explanations of Image Classifierscited by 15
  28. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  29. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  30. 2021Self-Supervised Learning of Audio Representations from Permutations with Differentiable Rankingcited by 14
  31. 2021Learning music audio representations via weak language supervisioncited by 13
  32. 2022ATST: Audio Representation Learning with Teacher-Student Transformercited by 13
  33. 2021Energy Consumption of Deep Generative Audio Modelscited by 12
  34. 2021Neural Waveshaping Synthesiscited by 12
  35. 2022Deep Performer: Score-to-Audio Music Performance Synthesiscited by 12
  36. 2022HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworkscited by 12
  37. 2024ReCLAP: Improving Zero Shot Audio Classification by Describing Soundscited by 12
  38. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12

50 most cited of 222 All 222 (JSON)