mlsanityData to 7 Oct 2026

Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompaniment

arXiv 2404.09313 ↗First posted 14 Apr 2024eess.AS cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

A song is a combination of singing voice and accompaniment. However, existing works focus on singing voice synthesis and music generation independently. Little attention was paid to explore song synthesis. In this work, we propose a novel task called text-to-song synthesis which incorporating both vocals and accompaniments generation. We develop Melodist, a two-stage text-to-song method that consists of singing voice synthesis (SVS) and vocal-to-accompaniment (V2A) synthesis. Melodist leverages tri-tower contrastive pretraining to learn more effective text representation for controllable V2A synthesis. A Chinese song dataset mined from a music website is built up to alleviate data scarcity for our research. The evaluation results on our dataset demonstrate that Melodist can synthesize songs with comparable quality and style consistency. Audio samples can be found in https://text2songMelodist.github.io/Sample/.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

33 of 41 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2021AST: Audio Spectrogram Transformercited by 634
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  10. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  11. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  12. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
21 more
  1. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  2. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  3. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  4. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  5. 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
  6. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  7. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  8. 2019Music Source Separation in the Waveform Domaincited by 98
  9. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  10. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  11. 2020PopMAG: Pop Music Accompaniment Generationcited by 68
  12. 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
  13. 2023SingSong: Generating musical accompaniments from singingcited by 54
  14. 2022Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesiscited by 53
  15. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  16. 2022Contrastive Audio-Language Learning for Musiccited by 32
  17. 2019Conditional LSTM-GAN for Melody Generation from Lyricscited by 31
  18. 2021SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generationcited by 27
  19. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  20. 2023Learning Music Sequence Representation from Text Supervisioncited by 5
  21. 2020Automatic Neural Lyrics and Melody Compositioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  2. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  3. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  4. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 21
  5. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
  6. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  7. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  8. 2026Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Controlcited by 7
  9. 2025YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidancecited by 5
  10. 2025Versatile Framework for Song Generation with Prompt-based Controlcited by 4
  11. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Trainingcited by 4
  12. 2024MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generationcited by 2
1 more
  1. 2026GCAgent: Enhancing Group Chat Communication through Dialogue Agents Systemcited by 0