# Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information

arXiv: https://arxiv.org/abs/2308.16836 · HTML: https://mlsanity.com/paper/2308.16836/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis](https://mlsanity.com/paper/2009.01776/) (2020)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [DeepSinger: Singing Voice Synthesis with Data Mined From the Web](https://mlsanity.com/paper/2007.04590/) (2020)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [Sequence-to-sequence Singing Synthesis Using the Feed-forward Transformer](https://mlsanity.com/paper/1910.09989/) (2019)
- [DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System](https://mlsanity.com/paper/2008.03009/) (2020)

## Built on by (most cited first, 4 of 4)

- [An End-to-End Approach for Chord-Conditioned Song Generation](https://mlsanity.com/paper/2409.06307/) (2024)
- [Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence](https://mlsanity.com/paper/2509.07038/) (2025)
- [Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training](https://mlsanity.com/paper/2309.00284/) (2023)
- [Polyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASR](https://mlsanity.com/paper/2603.16184/) (2026)

## Abstract (arXiv)

This paper presents an end-to-end high-quality singing voice synthesis (SVS) system that uses bidirectional encoder representation from Transformers (BERT) derived semantic embeddings to improve the expressiveness of the synthesized singing voice. Based on the main architecture of recently proposed VISinger, we put forward several specific designs for expressive singing voice synthesis. First, different from the previous SVS models, we use text representation of lyrics extracted from pre-trained BERT as additional input to the model. The representation contains information about semantics of the lyrics, which could help SVS system produce more expressive and natural voice. Second, we further introduce an energy predictor to stabilize the synthesized voice and model the wider range of energy variations that also contribute to the expressiveness of singing voice. Last but not the least, to attenuate the off-key issues, the pitch predictor is re-designed to predict the real to note pitch ratio. Both objective and subjective experimental results indicate that the proposed SVS system can produce singing voice with higher-quality outperforming VISinger.