mlsanityData to 7 Oct 2026

SongCreator: Lyrics-based Universal Song Generation

arXiv 2409.06029 ↗First posted 9 Sep 2024cs.SD cs.AI eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and a series of attention mask strategies for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various songrelated generation tasks by utilizing specific attention masks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different audio prompts, exhibiting its potential applicability. Our samples are available at https://thuhcsi.github.io/SongCreator/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 65 of its 75 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

65 of 75 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2021Variational Diffusion Modelscited by 1,145
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
38 more
  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  10. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  11. 2020Jukebox: A Generative Model for Musiccited by 535
  12. 2023Simple and Controllable Music Generationcited by 480
  13. 2023MusicLM: Generating Music From Textcited by 442
  14. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  15. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  16. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  17. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  18. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  19. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  20. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  21. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  22. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  23. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  24. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  25. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  26. 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
  27. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  28. 2022Hybrid Transformers for Music Source Separationcited by 134
  29. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  30. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  31. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  32. 2021Hybrid Spectrogram and Waveform Source Separationcited by 90
  33. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  34. 2023SingSong: Generating musical accompaniments from singingcited by 54
  35. 2023Efficient Neural Music Generationcited by 50
  36. 2023A Foundation Model for Music Informaticscited by 45
  37. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41
  38. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  2. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  3. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
  4. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 13
  5. 2025JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignmentcited by 9
  6. 2026Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Controlcited by 7
  7. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Trainingcited by 4
  8. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  9. 2025SLEEPING-DISCO 9M: A large-scale pre-training dataset for generative music modelingcited by 0
  10. 2025DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generationcited by 0
  11. 2026SegTune: Structured and Fine-Grained Control for Song Generationcited by 0
  12. 2026WanSong v1.0 Technical Reportcited by 0