mlsanityData to 19 Oct 2026

Adapting Speech Language Model to Singing Voice Synthesis

arXiv 2512.14657 ↗First posted 16 Dec 2025cs.SDcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Speech Language Models (SLMs) have recently emerged as a unified paradigm for addressing a wide range of speech-related tasks, including text-to-speech (TTS), speech enhancement (SE), and automatic speech recognition (ASR). However, the generalization capability of large-scale pre-trained SLMs remains underexplored. In this work, we adapt a 1.7B parameter TTS pretrained SLM for singing voice synthesis (SVS), using only a 135-hour synthetic singing corpus, ACE-Opencpop. Building upon the ESPNet-SpeechLM, our recipe involves the following procedure: (1) tokenization of music score conditions and singing waveforms, (2) multi-stream language model token prediction, (3) conditional flow matching-based mel-spectrogram generation. (4) a mel-to-wave vocoder. Experimental results demonstrate that our adapted SLM generalizes well to SVS and achieves performance comparable to leading discrete token-based SVS models.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 22 of its 27 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

22 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech24 Sep
  2. 2024TokSing: Singing Voice Synthesis based on Discrete Tokens12 Jun
  3. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision7 Feb
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  5. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 17

  1. 2016Density estimation using Real NVPcited by 2,092
  2. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,567
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  4. 2024Qwen2-Audio Technical Reportcited by 689
  5. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 563
  6. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 365
  7. 2025Kimi-Audio Technical Reportcited by 280
  8. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 75
  9. 2020XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis Systemcited by 41
  10. 2024MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Modelscited by 21
  11. 2024SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Predictioncited by 16
  12. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 14
5 more
  1. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 13
  2. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 13
  3. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 12
  4. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 12
  5. 2022Improved Normalizing Flow-Based Speech Enhancement using an All-pole Gammatone Filterbank for Conditional Input Representationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Systematic Exploration of Joint-training for Singing Voice Synthesis
  2. 2024LHQ-SVC: Lightweight and High Quality Singing Voice Conversion Modeling
  3. 2025Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation
  4. 2024Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion ↗
  5. 2021KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms
  6. 2022UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis
  7. 2022Karaoker: Alignment-free singing voice synthesis with speech training data
  8. 2024MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance