mlsanityData to 14 Oct 2026

StemGen: A music generation model that listens

arXiv 2312.08723 ↗First posted 14 Dec 2023cs.SD cs.LG eess.AScited by 32

Not on the map. Why?

Abstract (arXiv)

End-to-end generation of musical audio using deep learning techniques has seen an explosion of activity recently. However, most models concentrate on generating fully mixed music in response to abstract conditioning information. In this work, we present an alternative paradigm for producing music generation models that can listen and respond to musical context. We describe how such a model can be constructed using a non-autoregressive, transformer-based model architecture and present a number of novel architectural and sampling improvements. We train the described architecture on both an open-source and a proprietary dataset. We evaluate the produced models using standard quality metrics and a new approach based on music information retrieval descriptors. The resulting model reaches the audio quality of state-of-the-art text-conditioned models, as well as exhibiting strong musical coherence with its context.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

20 of 27 references.

References · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2020Jukebox: A Generative Model for Musiccited by 542
  6. 2023Simple and Controllable Music Generationcited by 488
  7. 2023MusicLM: Generating Music From Textcited by 448
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  9. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  10. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  11. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  12. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 78
8 more
  1. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  2. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  3. 2023SingSong: Generating musical accompaniments from singingcited by 55
  4. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  5. 2021SpecTNT: a Time-Frequency Transformer for Music Audiocited by 16
  6. 2023Multitrack Music Transcription with a Time-Frequency Perceivercited by 13
  7. 2022Modeling Beats and Downbeats with a Time-Frequency Transformercited by 12
  8. 2022To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functionscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 7; 2026: 9.

32 papers.

Sort:

By area:

  1. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  2. 2024Long-form music generation with latent diffusioncited by 89
  3. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  4. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  5. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  6. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  7. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  8. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 14
  9. 2024Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Modelcited by 13
  10. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  11. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  12. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
20 more
  1. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  2. 2024Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimationcited by 8
  3. 2026Stable Audio 3cited by 6
  4. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  5. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  6. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  7. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  8. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  9. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  10. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  11. 2026DARC: Drum accompaniment generation with fine-grained rhythm controlcited by 2
  12. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 2
  13. 2026Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSPcited by 2
  14. 2026LiveBand: Live Accompaniment Generation in the Audio Domaincited by 2
  15. 2024Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Modelscited by 1
  16. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  17. 2026Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidancecited by 1
  18. 2025BNMusic: Blending Environmental Noises into Personalized Musiccited by 0
  19. 2026SyncTrack: Rhythmic Stability and Synchronization in Multi-Track Music Generationcited by 0
  20. 2026Latent Fourier Transformcited by 0