# RMSSinger: Realistic-Music-Score based Singing Voice Synthesis

arXiv: https://arxiv.org/abs/2305.10686 · HTML: https://mlsanity.com/paper/2305.10686/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [PortaSpeech: Portable and High-Quality Generative Text-to-Speech](https://mlsanity.com/paper/2109.15166/) (2021)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System](https://mlsanity.com/paper/2006.06261/) (2020)
- [DeepSinger: Singing Voice Synthesis with Data Mined From the Web](https://mlsanity.com/paper/2007.04590/) (2020)
- [Revisiting Over-Smoothness in Text to Speech](https://mlsanity.com/paper/2202.13066/) (2022)
- [SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation](https://mlsanity.com/paper/2110.07468/) (2021)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech](https://mlsanity.com/paper/2205.07211/) (2022)
- [SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech](https://mlsanity.com/paper/2204.11792/) (2022)
- [WeSinger: Data-augmented Singing Voice Synthesis with Auxiliary Losses](https://mlsanity.com/paper/2203.10750/) (2022)

## Built on by (most cited first, 12 of 12)

- [DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion](https://mlsanity.com/paper/2503.01183/) (2025)
- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis](https://mlsanity.com/paper/2312.10741/) (2023)
- [TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control](https://mlsanity.com/paper/2409.15977/) (2024)
- [Accompanied Singing Voice Synthesis with Fully Text-controlled Melody](https://mlsanity.com/paper/2407.02049/) (2024)
- [RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text](https://mlsanity.com/paper/2405.20336/) (2024)
- [YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance](https://mlsanity.com/paper/2512.04779/) (2025)
- [Wav2SQL: Direct Generalizable Speech-To-SQL Parsing](https://mlsanity.com/paper/2305.12552/) (2023)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment](https://mlsanity.com/paper/2510.09016/) (2025)
- [Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence](https://mlsanity.com/paper/2509.07038/) (2025)
- [Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations](https://mlsanity.com/paper/2402.01520/) (2024)

## Abstract (arXiv)

We are interested in a challenging task, Realistic-Music-Score based Singing Voice Synthesis (RMS-SVS). RMS-SVS aims to generate high-quality singing voices given realistic music scores with different note types (grace, slur, rest, etc.). Though significant progress has been achieved, recent singing voice synthesis (SVS) methods are limited to fine-grained music scores, which require a complicated data collection pipeline with time-consuming manual annotation to align music notes with phonemes. Furthermore, these manual annotation destroys the regularity of note durations in music scores, making fine-grained music scores inconvenient for composing. To tackle these challenges, we propose RMSSinger, the first RMS-SVS method, which takes realistic music scores as input, eliminating most of the tedious manual annotation and avoiding the aforementioned inconvenience. Note that music scores are based on words rather than phonemes, in RMSSinger, we introduce word-level modeling to avoid the time-consuming phoneme duration annotation and the complicated phoneme-level mel-note alignment. Furthermore, we propose the first diffusion-based pitch modeling method, which ameliorates the naturalness of existing pitch-modeling methods. To achieve these, we collect a new dataset containing realistic music scores and singing voices according to these realistic music scores from professional singers. Extensive experiments on the dataset demonstrate the effectiveness of our methods. Audio samples are available at https://rmssinger.github.io/.