# Karaoker: Alignment-free singing voice synthesis with speech training data

arXiv: https://arxiv.org/abs/2204.04127 · HTML: https://mlsanity.com/paper/2204.04127/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 20 of 20)

- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect](https://mlsanity.com/paper/1803.01541/) (2018)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling](https://mlsanity.com/paper/2103.14574/) (2021)
- [Adversarially Trained End-to-end Korean Singing Voice Synthesis System](https://mlsanity.com/paper/1908.01919/) (2019)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency](https://mlsanity.com/paper/2111.09052/) (2021)
- [From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint](https://mlsanity.com/paper/2005.04587/) (2020)
- [Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer](https://mlsanity.com/paper/2006.10317/) (2020)
- [Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities](https://mlsanity.com/paper/1909.10302/) (2019)
- [Singing Synthesis: with a little help from my attention](https://mlsanity.com/paper/1912.05881/) (2019)
- [A Melody-Unsupervision Model for Singing Voice Synthesis](https://mlsanity.com/paper/2110.06546/) (2021)
- [Fast DCTTS: Efficient Deep Convolutional Text-to-Speech](https://mlsanity.com/paper/2104.00624/) (2021)
- [Singing-Tacotron: Global duration control attention and dynamic filter for End-to-end singing voice synthesis](https://mlsanity.com/paper/2202.07907/) (2022)
- [A Comparative Study of Pitch Extraction Algorithms on a Large Variety of Singing Sounds](https://mlsanity.com/paper/1912.12609/) (2019)
- [Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.10168/) (2021)
- [Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.09146/) (2021)

## Built on by (most cited first, 1 of 1)

- [Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations](https://mlsanity.com/paper/2402.01520/) (2024)

## Abstract (arXiv)

Existing singing voice synthesis models (SVS) are usually trained on singing data and depend on either error-prone time-alignment and duration features or explicit music score information. In this paper, we propose Karaoker, a multispeaker Tacotron-based model conditioned on voice characteristic features that is trained exclusively on spoken data without requiring time-alignments. Karaoker synthesizes singing voice and transfers style following a multi-dimensional template extracted from a source waveform of an unseen singer/speaker. The model is jointly conditioned with a single deep convolutional encoder on continuous data including pitch, intensity, harmonicity, formants, cepstral peak prominence and octaves. We extend the text-to-speech training objective with feature reconstruction, classification and speaker identification tasks that guide the model to an accurate result. In addition to multitasking, we also employ a Wasserstein GAN training scheme as well as new losses on the acoustic model's output to further refine the quality of the model.