mlsanityData to 14 Oct 2026

Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls

arXiv 2307.10304 ↗First posted 19 Jul 2023cs.SD cs.LG eess.AScited by 35

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose Polyffusion, a diffusion model that generates polyphonic music scores by regarding music as image-like piano roll representations. The model is capable of controllable music generation with two paradigms: internal control and external control. Internal control refers to the process in which users pre-define a part of the music and then let the model infill the rest, similar to the task of masked music generation (or music inpainting). External control conditions the model with external yet related information, such as chord, texture, or other features, via the cross-attention mechanism. We show that by using internal and external controls, Polyffusion unifies a wide range of music creation tasks, including melody generation given accompaniment, accompaniment generation given melody, arbitrary music segment inpainting, and music arrangement given chords or textures. Experimental results show that our model significantly outperforms existing Transformer and sampling-based baselines, and using pre-trained disentangled representations as external conditions yields more effective controls.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 42 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

35 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability11 Oct
  2. 2021Symbolic Music Generation with Diffusion Models30 Mar
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2022Video Diffusion Modelscited by 2,305
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  12. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
20 more
  1. 2020Jukebox: A Generative Model for Musiccited by 542
  2. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  3. 2023MusicLM: Generating Music From Textcited by 448
  4. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  5. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  6. 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
  7. 2016DeepBach: a Steerable Model for Bach Chorales Generationcited by 91
  8. 2020POP909: A Pop-song Dataset for Music Arrangement Generationcited by 83
  9. 2020PopMAG: Pop Music Accompaniment Generationcited by 68
  10. 2019LakhNES: Improving multi-instrumental music generation with cross-domain pre-trainingcited by 54
  11. 2019Counterpoint by Convolutioncited by 43
  12. 2020Learning Interpretable Representation for Controllable Polyphonic Music Generationcited by 40
  13. 2020Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythmcited by 28
  14. 2017Music Generation by Deep Learning - Challenges and Directionscited by 24
  15. 2021AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfercited by 22
  16. 2021TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Methodcited by 20
  17. 2022MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Controlcited by 17
  18. 2018Music Style Transfer: A Position Papercited by 16
  19. 2020Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositionscited by 16
  20. 2021Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encodingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 11; 2026: 10.

35 papers.

Sort:

By area:

  1. 2024Symbolic Music Generation with Non-Differentiable Rule Guided Diffusioncited by 54
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  4. 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Modelscited by 25
  5. 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
  6. 2023Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editingcited by 15
  7. 2024DExter: Learning and Controlling Performance Expression with Diffusion Modelscited by 11
  8. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
  9. 2023Structured Multi-Track Accompaniment Arrangement via Style Prior Modellingcited by 9
  10. 2024Composer's Assistant 2: Interactive Multi-Track MIDI Infilling with Fine-Grained User Controlcited by 5
  11. 2025Vision-to-Music Generation: A Surveycited by 4
  12. 2025Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generationcited by 4
23 more
  1. 2025ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinementcited by 3
  2. 2026BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Stepscited by 3
  3. 2024SYMPLEX: Controllable Symbolic Music Generation using Simplex Diffusion with Vocabulary Priorscited by 2
  4. 2025Adaptable Symbolic Music Infilling with MIDI-RWKVcited by 2
  5. 2025Diffusion-based Symbolic Music Generation with Structured State Space Modelscited by 2
  6. 2025Segment-Factorized Full-Song Generation on Symbolic Piano Musiccited by 2
  7. 2025PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generationcited by 2
  8. 2024Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generationcited by 1
  9. 2026Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generationcited by 1
  10. 2026ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Modelscited by 1
  11. 2026BeatEdit: Symbolic Music Generation as Explicit Editingcited by 1
  12. 2024Pictures Of MIDI: Controlled Music Generation via Graphical Prompts for Image-Based Diffusion Inpaintingcited by 0
  13. 2024Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatisecited by 0
  14. 2025TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structurecited by 0
  15. 2025WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restorationcited by 0
  16. 2025Visually Grounded Narratives: Reducing Cognitive Burden in Researcher-Participant Interactioncited by 0
  17. 2025Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generationcited by 0
  18. 2026Pay (Cross) Attention to the Melody: Curriculum Masking for Single-Encoder Melodic Harmonizationcited by 0
  19. 2026D3PIA: A Discrete Denoising Diffusion Model for Piano Accompaniment Generation From Lead sheetcited by 0
  20. 2026Structure-Aware Piano Accompaniment via Style Planning and Dataset-Aligned Pattern Retrievalcited by 0
  21. 2026PianoKontext: Expressive Performance Rendering from Deadpan Contextcited by 0
  22. 2026MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Musiccited by 0
  23. 2026Diff-Symbo: Text-Controlled Long-Duration Symbolic Music Generation Using Autoregressive Latent Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffuseRoll: Multi-track multi-category music generation based on diffusion model
  2. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  3. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models
  4. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  5. 2023Controllable Music Production with Diffusion Models and Guidance Gradients
  6. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models
  7. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  8. 2025Similarity-Guided Diffusion for Long-Gap Music Inpainting