mlsanityData to 7 Oct 2026

JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models

arXiv 2308.04729 ↗First posted 9 Aug 2023cs.SD cs.AI cs.LGcited by 41

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Music generation has attracted growing interest with the advancement of deep generative models. However, generating music conditioned on textual descriptions, known as text-to-music, remains challenging due to the complexity of musical structures and high sampling rate requirements. Despite the task's significance, prevailing generative models exhibit limitations in music quality, computational efficiency, and generalization. This paper introduces JEN-1, a universal high-fidelity model for text-to-music generation. JEN-1 is a diffusion model incorporating both autoregressive and non-autoregressive training. Through in-context learning, JEN-1 performs various generation tasks including text-guided music generation, music inpainting, and continuation. Evaluations demonstrate JEN-1's superior performance over state-of-the-art methods in text-music alignment and music quality while maintaining computational efficiency. Our demos are available at https://jenmusic.ai/audio-demos

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 45 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

35 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  3. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  4. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  5. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion27 Jan
  6. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
16 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2017Generative Adversarial Networks: An Overviewcited by 588
  4. 2020Jukebox: A Generative Model for Musiccited by 535
  5. 2023Simple and Controllable Music Generationcited by 480
  6. 2023MusicLM: Generating Music From Textcited by 442
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  8. 2022AudioGen: Textually Guided Audio Generationcited by 292
  9. 2019Self-Correction for Human Parsingcited by 179
  10. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  11. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  12. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 47
  13. 2019Conditional LSTM-GAN for Melody Generation from Lyricscited by 31
  14. 2022Symbolic music generation conditioned on continuous-valued emotionscited by 12
  15. 2022In-N-Out Generative Learning for Dense Unsupervised Video Segmentationcited by 7
  16. 2021Super-Resolving Cross-Domain Face Miniatures by Peeking at One-Shot Exemplarcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 23; 2025: 10; 2026: 4.

41 papers.

Sort:

By area:

  1. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  2. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  3. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  4. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  5. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  6. 2024Foundation Models for Music: A Surveycited by 34
  7. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  8. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  9. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
  10. 2024FLUX that Plays Musiccited by 18
  11. 2023JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generationcited by 17
  12. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
29 more
  1. 2024Bass Accompaniment Generation via Latent Diffusioncited by 16
  2. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  3. 2024MusicFlow: Cascaded Flow Matching for Text Guided Music Generationcited by 14
  4. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  5. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  6. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 13
  7. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  8. 2024MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generationcited by 11
  9. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  10. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 10
  11. 2024Music Consistency Modelscited by 7
  12. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generationcited by 6
  13. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  14. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  15. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  16. 2025MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generationcited by 5
  17. 2026Stable Audio 3cited by 5
  18. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  19. 2025Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Modelscited by 4
  20. 2025NOTA: Multimodal Music Notation Understanding for Visual Large Language Modelcited by 4
  21. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  22. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generationcited by 2
  23. 2025Exploring State-Space-Model based Language Model in Music Generationcited by 2
  24. 2023Can MusicGen Create Training Data for MIR Tasks?cited by 1
  25. 2024JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuningcited by 1
  26. 2024Towards Training Music Taggers on Synthetic Datacited by 1
  27. 2026JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitionscited by 0
  28. 2026MusicMark: A Robust Generative Watermarking Framework for Music Generationcited by 0
  29. 2026MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  2. 2026S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation
  3. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  4. 2026Training-Efficient Text-to-Music Generation with State-Space Modeling
  5. 2024Long-form music generation with latent diffusion
  6. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗
  7. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
  8. 2023Efficient Neural Music Generation