# SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation

arXiv: https://arxiv.org/abs/2502.13128 · HTML: https://mlsanity.com/paper/2502.13128/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec](https://mlsanity.com/paper/2305.02765/) (2023)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)

## Built on by (most cited first, 29 of 29)

- [SIM-CoT: Supervised Implicit Chain-of-Thought](https://mlsanity.com/paper/2509.20317/) (2025)
- [CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning](https://mlsanity.com/paper/2509.22647/) (2025)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching](https://mlsanity.com/paper/2510.22950/) (2025)
- [Live Music Models](https://mlsanity.com/paper/2508.04651/) (2025)
- [JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment](https://mlsanity.com/paper/2507.20880/) (2025)
- [Music Arena: Live Evaluation for Text-to-Music](https://mlsanity.com/paper/2507.20900/) (2025)
- [Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control](https://mlsanity.com/paper/2601.03973/) (2026)
- [LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training](https://mlsanity.com/paper/2606.30642/) (2026)
- [Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing](https://mlsanity.com/paper/2602.16455/) (2026)
- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [TADA! Tuning Audio Diffusion Models through Activation Steering](https://mlsanity.com/paper/2602.11910/) (2026)
- [Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notation](https://mlsanity.com/paper/2508.01394/) (2025)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation](https://mlsanity.com/paper/2604.17656/) (2026)
- [UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions](https://mlsanity.com/paper/2604.22209/) (2026)
- [Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation](https://mlsanity.com/paper/2508.05011/) (2025)
- [DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generation](https://mlsanity.com/paper/2511.20224/) (2025)
- [Echoes: A semantically-aligned music deepfake detection dataset](https://mlsanity.com/paper/2603.23667/) (2026)
- [LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation](https://mlsanity.com/paper/2604.11052/) (2026)
- [SegTune: Structured and Fine-Grained Control for Song Generation](https://mlsanity.com/paper/2606.02638/) (2026)
- [Exploring LLMs for South Asian Music Understanding and Generation](https://mlsanity.com/paper/2606.05522/) (2026)
- [Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation](https://mlsanity.com/paper/2606.24307/) (2026)
- [Noisy-Channel Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2607.05198/) (2026)
- [MusicMark: A Robust Generative Watermarking Framework for Music Generation](https://mlsanity.com/paper/2607.11117/) (2026)
- [WanSong v1.0 Technical Report](https://mlsanity.com/paper/2607.14749/) (2026)
- [AURA: Unified Multimodal Framework for Conversational Music Editing](https://mlsanity.com/paper/2609.14344/) (2026)
- [Ghost in the Encoder: Decodable Artist Identity Representations in Lyrics-to-Song Generation](https://mlsanity.com/paper/2609.39552/) (2026)

## Abstract (arXiv)

Text-to-song generation, the task of creating vocals and accompaniment from textual inputs, poses significant challenges due to domain complexity and data scarcity. Existing approaches often employ multi-stage generation procedures, leading to cumbersome training and inference pipelines, as well as suboptimal overall generation quality due to error accumulation across stages. In this paper, we propose SongGen, a fully open-source, single-stage auto-regressive transformer designed for controllable song generation. The proposed model facilitates fine-grained control over diverse musical attributes, including lyrics and textual descriptions of instrumentation, genre, mood, and timbre, while also offering an optional three-second reference clip for voice cloning. Within a unified auto-regressive framework, SongGen supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately for greater flexibility in downstream applications. We explore diverse token pattern strategies for each mode, leading to notable improvements and valuable insights. Furthermore, we design an automated data preprocessing pipeline with effective quality control. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline. The code is available at https://github.com/LiuZH-19/SongGen.