# Singing Voice Synthesis Based on a Musical Note Position-Aware Attention Mechanism

arXiv: https://arxiv.org/abs/2212.13703 · HTML: https://mlsanity.com/paper/2212.13703/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 28 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis](https://mlsanity.com/paper/2009.01776/) (2020)
- [Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS](https://mlsanity.com/paper/1906.00672/) (2019)
- [XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System](https://mlsanity.com/paper/2006.06261/) (2020)
- [Adversarially Trained End-to-end Korean Singing Voice Synthesis System](https://mlsanity.com/paper/1908.01919/) (2019)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis](https://mlsanity.com/paper/1807.06736/) (2018)
- [Sequence-to-sequence Singing Synthesis Using the Feed-forward Transformer](https://mlsanity.com/paper/1910.09989/) (2019)
- [A Neural Parametric Singing Synthesizer](https://mlsanity.com/paper/1704.03809/) (2017)
- [Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System](https://mlsanity.com/paper/2108.02776/) (2021)
- [Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer](https://mlsanity.com/paper/2006.10317/) (2020)
- [Singing Synthesis: with a little help from my attention](https://mlsanity.com/paper/1912.05881/) (2019)
- [Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling](https://mlsanity.com/paper/1906.08977/) (2019)
- [Fast and High-Quality Singing Voice Synthesis System based on Convolutional Neural Networks](https://mlsanity.com/paper/1910.11690/) (2019)
- [Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss](https://mlsanity.com/paper/2010.12024/) (2020)
- [Singing-Tacotron: Global duration control attention and dynamic filter for End-to-end singing voice synthesis](https://mlsanity.com/paper/2202.07907/) (2022)
- [Neural Sequence-to-Sequence Speech Synthesis Using a Hidden Semi-Markov Model Based Structured Attention Mechanism](https://mlsanity.com/paper/2108.13985/) (2021)

## Abstract (arXiv)

This paper proposes a novel sequence-to-sequence (seq2seq) model with a musical note position-aware attention mechanism for singing voice synthesis (SVS). A seq2seq modeling approach that can simultaneously perform acoustic and temporal modeling is attractive. However, due to the difficulty of the temporal modeling of singing voices, many recent SVS systems with an encoder-decoder-based model still rely on explicitly on duration information generated by additional modules. Although some studies perform simultaneous modeling using seq2seq models with an attention mechanism, they have insufficient robustness against temporal modeling. The proposed attention mechanism is designed to estimate the attention weights by considering the rhythm given by the musical score. Furthermore, several techniques are also introduced to improve the modeling performance of the singing voice. Experimental results indicated that the proposed model is effective in terms of both naturalness and robustness of timing.