mlsanityData to 7 Oct 2026

FLUX that Plays Music

arXiv 2409.00587 ↗First posted 1 Sep 2024cs.SD cs.CV eess.AScited by 18

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper explores a simple extension of diffusion-based rectified flow Transformers for text-to-music generation, termed as FluxMusic. Generally, along with design in advanced Fluxfootnotehttps://github.com/black-forest-labs/flux model, we transfers it into a latent VAE space of mel-spectrum. It involves first applying a sequence of independent attention to the double text-music stream, followed by a stacked single music stream for denoised patch prediction. We employ multiple pre-trained text encoders to sufficiently capture caption semantic information as well as inference flexibility. In between, coarse textual information, in conjunction with time step embeddings, is utilized in a modulation mechanism, while fine-grained textual details are concatenated with the music patch sequence as inputs. Through an in-depth study, we demonstrate that rectified flow training with an optimized architecture significantly outperforms established diffusion methods for the text-to-music task, as evidenced by various automatic metrics and human preference evaluations. Our experimental data, code, and model weights are made publicly available at: urlhttps://github.com/feizc/FluxMusic.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 72 of its 89 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

72 of 89 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generation24 May
  2. 2023Mustango: Toward Controllable Text-to-Music Generation14 Nov
  3. 2023Investigating Personalization Methods in Text to Music Generation20 Sep
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  5. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models9 Aug
  6. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies3 Aug
  7. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  8. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer22 May
  9. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  10. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  11. 2021Symbolic Music Generation with Diffusion Models30 Mar
  12. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2023Consistency Modelscited by 1,778
  12. 2020Point Transformercited by 1,462
  13. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  14. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  15. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  16. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  17. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  18. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  19. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  20. 2022High Fidelity Neural Audio Compressioncited by 673
  21. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  22. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  23. 2023Simple and Controllable Music Generationcited by 480
  24. 2023MusicLM: Generating Music From Textcited by 442
  25. 2023Improved Techniques for Training Consistency Modelscited by 430
  26. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  27. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  28. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  29. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 276
  30. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  31. 2016FMA: A Dataset For Music Analysiscited by 205
  32. 2024Stable Audio Opencited by 192
  33. 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
  34. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  35. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  36. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
  37. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  38. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  3. 2025SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformerscited by 33
  4. 2025MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigmcited by 30
  5. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  6. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  7. 2024AMO Sampler: Enhancing Text Rendering with Overshootingcited by 11
  8. 2024Video Diffusion Transformers are In-Context Learnerscited by 11
  9. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  10. 2025Improving Rectified Flow with Boundary Conditionscited by 9
  11. 2025SonicMaster: Towards Controllable All-in-One Music Restoration and Masteringcited by 6
  12. 2024Token Turing Machines are Efficient Vision Modelscited by 2
6 more
  1. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  2. 2025BemaGANv2: Discriminator Combination Strategies for GAN-based Vocoders in Long-Term Audio Generationcited by 0
  3. 2025TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantizationcited by 0
  4. 2025MusRec: Zero-Shot Text-to-Music Editing via Rectified Flow and Diffusion Transformerscited by 0
  5. 2026Improving Text-to-Music Generation with Human Preference Rewardscited by 0
  6. 2026FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models
  2. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  3. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning
  4. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer
  5. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformers
  6. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
  7. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  8. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗