mlsanityData to 14 Oct 2026

Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models

arXiv 2406.08384 ↗First posted 12 Jun 2024cs.SD cs.AI eess.AScited by 33

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advancements in deep generative models present new opportunities for music production but also pose challenges, such as high computational demands and limited audio quality. Moreover, current systems frequently rely solely on text input and typically focus on producing complete musical pieces, which is incompatible with existing workflows in music production. To address these issues, we introduce "Diff-A-Riff," a Latent Diffusion Model designed to generate high-quality instrumental accompaniments adaptable to any musical context. This model offers control through either audio references, text prompts, or both, and produces 48kHz pseudo-stereo audio while significantly reducing inference time and memory usage. We demonstrate the model's capabilities through objective metrics and subjective listening tests, with extensive examples available on the accompanying website: sonycslparis.github.io/diffariff-companion/

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 45 of its 56 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

45 of 56 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models18 Mar
  2. 2024Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion15 Feb
  3. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models9 Feb
  4. 2024Bass Accompaniment Generation via Latent Diffusion2 Feb
  5. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generation22 Jan
  6. 2023EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis15 Nov
  7. 2023Controllable Music Production with Diffusion Models and Guidance Gradients1 Nov
  8. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  9. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models9 Aug
  10. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  11. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation4 Feb
  12. 2023ArchiSound: Audio Generation with Diffusion30 Jan
2 more
  1. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  2. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun

Other references · 31

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2013Auto-Encoding Variational Bayescited by 4,053
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2023Consistency Modelscited by 1,822
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  11. 2018Demystifying MMD GANscited by 1,293
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
19 more
  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2020Jukebox: A Generative Model for Musiccited by 542
  3. 2023Simple and Controllable Music Generationcited by 488
  4. 2023MusicLM: Generating Music From Textcited by 448
  5. 2023Improved Techniques for Training Consistency Modelscited by 439
  6. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  7. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  8. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  9. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 83
  10. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 73
  11. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  12. 2023SingSong: Generating musical accompaniments from singingcited by 55
  13. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  14. 2020DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networkscited by 35
  15. 2023StemGen: A music generation model that listenscited by 32
  16. 2022Musika! Fast Infinite Waveform Music Generationcited by 24
  17. 2019High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interactioncited by 14
  18. 2022JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAEcited by 11
  19. 2022DrumGAN VST: A Plugin for Drum Sound Analysis/Synthesis With Autoencoding Generative Adversarial Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 14; 2026: 10.

33 papers.

Sort:

By area:

  1. 2024Foundation Models for Music: A Surveycited by 37
  2. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  3. 2025Aligning Text-to-Music Evaluation with Human Preferencescited by 21
  4. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  5. 2025Fast Text-to-Audio Generation with Adversarial Post-Trainingcited by 14
  6. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  7. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  8. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  9. 2024Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimationcited by 8
  10. 2025MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AIcited by 7
  11. 2025Steering Autoregressive Music Generation with Recursive Feature Machinescited by 6
  12. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
21 more
  1. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  2. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  3. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  4. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  5. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpaintingcited by 3
  6. 2024Subtractive Training for Music Stem Insertion using Latent Diffusion Modelscited by 2
  7. 2025Latent Granular Resynthesis using Neural Audio Codecscited by 2
  8. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 2
  9. 2026Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSPcited by 2
  10. 2026LiveBand: Live Accompaniment Generation in the Audio Domaincited by 2
  11. 2024The Interpretation Gap in Text-to-Music Generation Modelscited by 1
  12. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  13. 2025Learning Linearity in Audio Consistency Autoencoders via Implicit Regularizationcited by 1
  14. 2026Guiding Token-Sparse Diffusion Modelscited by 1
  15. 2026Towards Effective Negation Modeling in Joint Audio-Text Models for Musiccited by 1
  16. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  17. 2025LoopGen: Training-Free Loopable Music Generationcited by 0
  18. 2025Learning Relationships Between Separate Audio Tracks for Creative Applicationscited by 0
  19. 2026Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separationcited by 0
  20. 2026JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitionscited by 0
  21. 2026How Much AI Is in This Track? Quantifying the Proportion of AI-Generated Stems in Hybrid Music Mixturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
  2. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  3. 2024Long-form music generation with latent diffusion
  4. 2024Combining audio control and style transfer using latent diffusion
  5. 2026Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
  6. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  7. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  8. 2023Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls