mlsanityData to 7 Oct 2026

Mousai: Text-to-Music Generation with Long-Context Latent Diffusion

arXiv 2301.11757 ↗First posted 27 Jan 2023cs.CL cs.LG cs.SDcited by 77

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent years have seen the rapid development of large generative models for text; however, much less research has explored the connection between text and another "language" of communication -- music. Music, much like text, can convey emotions, stories, and ideas, and has its own unique structure and syntax. In our work, we bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Mousai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. Moreover, our model features high efficiency, which enables real-time inference on a single consumer GPU with a reasonable speed. Through experiments and property analyses, we show our model's competence over a variety of criteria compared with existing music generation models. Lastly, to promote the open-source culture, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: Codes: https://github.com/archinetai/audio-diffusion-pytorch; music samples for this paper: http://bit.ly/44ozWDH; all music samples for all models: https://bit.ly/audio-diffusion.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 13 of its 38 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

13 of 38 references.

References · 13

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2020Jukebox: A Generative Model for Musiccited by 535
  9. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  10. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  11. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  12. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
1 more
  1. 2022ComMU: Dataset for Combinatorial Music Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 36; 2025: 17; 2026: 5.

77 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 480
  2. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  3. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  4. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  5. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  6. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  7. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  8. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  9. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  10. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  11. 2024Foundation Models for Music: A Surveycited by 34
  12. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
38 more
  1. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compressioncited by 33
  2. 2023StemGen: A music generation model that listenscited by 32
  3. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  4. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  5. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  6. 2023Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approachcited by 21
  7. 2023Content-based Controls For Music Large Language Modelingcited by 19
  8. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  9. 2024UniMuMo: Unified Text, Music and Motion Generationcited by 19
  10. 2023Long-Term Rhythmic Video Soundtrackercited by 17
  11. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 17
  12. 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
  13. 2023DiffEnc: Variational Diffusion with a Learned Encodercited by 16
  14. 2024Bass Accompaniment Generation via Latent Diffusioncited by 16
  15. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  16. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  17. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  18. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
  19. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Modelscited by 13
  20. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  21. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  22. 2023LongDanceDiff: Long-term Dance Generation with Conditional Diffusion Modelcited by 12
  23. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  24. 2025ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusioncited by 11
  25. 2024Music Style Transfer with Time-Varying Inversion of Diffusion Modelscited by 9
  26. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  27. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  28. 2024Music Consistency Modelscited by 7
  29. 2024Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Modelscited by 7
  30. 2024RapVerse: Coherent Vocals and Whole-Body Motions Generations from Textcited by 7
  31. 2025CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audiocited by 7
  32. 2023Zero-Shot Duet Singing Voices Separation with Diffusion Modelscited by 6
  33. 2024ByteComposer: a Human-like Melody Composition Method based on Language Model Agentcited by 6
  34. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generationcited by 6
  35. 2024Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintingscited by 6
  36. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  37. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  38. 2024FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generationcited by 4

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Long-form music generation with latent diffusion
  2. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  3. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
  4. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models
  6. 2023ArchiSound: Audio Generation with Diffusion
  7. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
  8. 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models