# Learning the Beauty in Songs: Neural Singing Voice Beautifier

arXiv: https://arxiv.org/abs/2202.13277 · HTML: https://mlsanity.com/paper/2202.13277/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [A Wavenet for Speech Denoising](https://mlsanity.com/paper/1706.07162/) (2017)
- [CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion](https://mlsanity.com/paper/1904.04631/) (2019)
- [DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion](https://mlsanity.com/paper/2105.13871/) (2021)
- [XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System](https://mlsanity.com/paper/2006.06261/) (2020)
- [DeepSinger: Singing Voice Synthesis with Data Mined From the Web](https://mlsanity.com/paper/2007.04590/) (2020)
- [Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning](https://mlsanity.com/paper/2103.09420/) (2021)
- [Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion](https://mlsanity.com/paper/1906.00794/) (2019)
- [Adversarially Trained End-to-end Korean Singing Voice Synthesis System](https://mlsanity.com/paper/1908.01919/) (2019)
- [Sequence-to-sequence Singing Synthesis Using the Feed-forward Transformer](https://mlsanity.com/paper/1910.09989/) (2019)
- [DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System](https://mlsanity.com/paper/2008.03009/) (2020)
- [Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer](https://mlsanity.com/paper/2006.10317/) (2020)
- [PPG-based singing voice conversion with adversarial representation learning](https://mlsanity.com/paper/2010.14804/) (2020)
- [Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding](https://mlsanity.com/paper/2110.04754/) (2021)
- [KaraTuner: Towards end to end natural pitch correction for singing voice in karaoke](https://mlsanity.com/paper/2110.09121/) (2021)
- [Singing voice correction using canonical time warping](https://mlsanity.com/paper/1711.08600/) (2017)
- [Deep Autotuner: a Pitch Correcting Network for Singing Performances](https://mlsanity.com/paper/2002.05511/) (2020)

## Built on by (most cited first, 10 of 10)

- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control](https://mlsanity.com/paper/2409.15977/) (2024)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis](https://mlsanity.com/paper/2505.14910/) (2025)
- [YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases](https://mlsanity.com/paper/2512.04793/) (2025)
- [AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment](https://mlsanity.com/paper/2305.04476/) (2023)
- [R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversion](https://mlsanity.com/paper/2510.20677/) (2025)
- [BERT-APC: A Reference-free Framework for Automatic Pitch Correction via Musical Context Inference](https://mlsanity.com/paper/2511.20006/) (2025)

## Abstract (arXiv)

We are interested in a novel task, singing voice beautifying (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are immature, and most of them are restricted to intonation but ignore the overall aesthetic quality. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping approach for pitch correction: Shape-Aware Dynamic Time Warping (SADTW), which ameliorates the robustness of existing time-warping approaches, to synchronize the amateur recording with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. To achieve this, we also propose a new dataset containing parallel singing recordings of both amateur and professional versions. Extensive experiments on both Chinese and English songs demonstrate the effectiveness of our methods in terms of both objective and subjective metrics. Audio samples are available at~urlhttps://neuralsvb.github.io. Codes: urlhttps://github.com/MoonInTheRiver/NeuralSVB.