mlsanityData to 13 Oct 2026

VampNet: Music Generation via Masked Acoustic Token Modeling

arXiv 2307.04686 ↗First posted 10 Jul 2023cs.SD cs.AI eess.AScited by 57

Not on the map. Why?

Abstract (arXiv)

We introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which allows us to sample coherent music from the model by applying a variety of masking approaches (called prompts) during inference. VampNet is non-autoregressive, leveraging a bidirectional transformer architecture that attends to all tokens in a forward pass. With just 36 sampling passes, VampNet can generate coherent high-fidelity musical waveforms. We show that by prompting VampNet in various ways, we can apply it to tasks like music compression, inpainting, outpainting, continuation, and looping with variation (vamping). Appropriately prompted, VampNet is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music. This flexible prompting capability makes VampNet a powerful music co-creation tool. Code and audio samples are available online.

Key facts. Cited by 57 papers in the map (data through 13 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  7. 2017Neural Discrete Representation Learningcited by 4,680
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,077
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  10. 2016Improved Techniques for Training GANscited by 2,946
  11. 2017Decoupled Weight Decay Regularizationcited by 1,788
  12. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,685
17 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,116
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,090
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 745
  5. 2022High Fidelity Neural Audio Compressioncited by 680
  6. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 641
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 613
  8. 2020Jukebox: A Generative Model for Musiccited by 537
  9. 2023Simple and Controllable Music Generationcited by 484
  10. 2023MusicLM: Generating Music From Textcited by 445
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 427
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 342
  13. 2022Efficient Training of Language Models to Fill in the Middlecited by 251
  14. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  15. 2023SoundStorm: Efficient Parallel Audio Generationcited by 113
  16. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 48
  17. 2021WaveBeat: End-to-end beat and downbeat tracking in the time domaincited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 34; 2025: 14; 2026: 7.

57 papers.

Sort:

By area:

  1. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 180
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  3. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  4. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  5. 2024Long-form music generation with latent diffusioncited by 89
  6. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  7. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  8. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  9. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  10. 2024Foundation Models for Music: A Surveycited by 35
  11. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  12. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
38 more
  1. 2023StemGen: A music generation model that listenscited by 32
  2. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  3. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  4. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  5. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  6. 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
  7. 2023Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editingcited by 15
  8. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  9. 2024Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Modelcited by 13
  10. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Modelscited by 13
  11. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  12. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 11
  13. 2024Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audiocited by 10
  14. 2025Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusioncited by 10
  15. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  16. 2024Detecting music deepfakes is easy but actually hardcited by 7
  17. 2025MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AIcited by 7
  18. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  19. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modelingcited by 6
  20. 2026Stable Audio 3cited by 6
  21. 2024Efficient Parallel Audio Generation using Group Masked Language Modelingcited by 5
  22. 2025Learning to Upsample and Upmix Audio in the Latent Domaincited by 5
  23. 2025A Comprehensive Survey on Generative AI for Video-to-Music Generationcited by 4
  24. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  25. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  26. 2025Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoderscited by 4
  27. 2024Sound Check: Auditing Audio Datasetscited by 3
  28. 2024Subtractive Training for Music Stem Insertion using Latent Diffusion Modelscited by 2
  29. 2024Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraintscited by 2
  30. 2024Watermarking Training Data of Music Generation Modelscited by 2
  31. 2024Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Modelscited by 1
  32. 2024Six Dragons Fly Again: Reviving 15th-Century Korean Court Music with Transformers and Novel Encodingcited by 1
  33. 2024Prevailing Research Areas for Music AI in the Era of Foundation Modelscited by 1
  34. 2024Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillationcited by 1
  35. 2025WhAM: Towards A Translative Model of Sperm Whale Vocalizationcited by 1
  36. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generatorscited by 1
  37. 2024LVNS-RAVE: Diversified audio generation with RAVE and Latent Vector Novelty Searchcited by 0
  38. 2025HARP 2.0: Expanding Hosted, Asynchronous, Remote Processing for Deep Learning in the DAWcited by 0

50 most cited of 57 All 57 (JSON)