# Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence

arXiv: https://arxiv.org/abs/2509.07038 · HTML: https://mlsanity.com/paper/2509.07038/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [Music ControlNet: Multiple Time-varying Controls for Music Generation](https://mlsanity.com/paper/2311.07069/) (2023)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching](https://mlsanity.com/paper/2502.12572/) (2025)
- [RMSSinger: Realistic-Music-Score based Singing Voice Synthesis](https://mlsanity.com/paper/2305.10686/) (2023)
- [Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt](https://mlsanity.com/paper/2403.11780/) (2024)
- [Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation](https://mlsanity.com/paper/2211.00996/) (2022)
- [Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information](https://mlsanity.com/paper/2308.16836/) (2023)
- [MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance](https://mlsanity.com/paper/2406.05965/) (2024)
- [Period Singer: Integrating Periodic and Aperiodic Variational Autoencoders for Natural-Sounding End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2406.09894/) (2024)
- [SiFiSinger: A High-Fidelity End-to-End Singing Voice Synthesizer based on Source-filter Model](https://mlsanity.com/paper/2410.12536/) (2024)

## Built on by (most cited first, 1 of 1)

- [Vibrato Expression Control for Singing Voice Conversion with Improving Independent Control](https://mlsanity.com/paper/2606.17126/) (2026)

## Abstract (arXiv)

Controllable Singing Voice Synthesis (SVS) aims to generate expressive singing voices reflecting user intent. While recent SVS systems achieve high audio quality, most rely on probabilistic modeling, limiting precise control over attributes such as dynamics. We address this by focusing on dynamic control--temporal loudness variation essential for musical expressiveness--and explicitly condition the SVS model on energy sequences extracted from ground-truth spectrograms, reducing annotation costs and improving controllability. We also propose a phoneme-level energy sequence for user-friendly control. To the best of our knowledge, this is the first attempt enabling user-driven dynamics control in SVS. Experiments show our method achieves over 50% reduction in mean absolute error of energy sequences for phoneme-level inputs compared to baseline and energy-predictor models, without compromising synthesis quality.