# LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation

arXiv: https://arxiv.org/abs/2408.12354 · HTML: https://mlsanity.com/paper/2408.12354/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 24 of its 26 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion](https://mlsanity.com/paper/2105.13871/) (2021)
- [Unsupervised Cross-Domain Singing Voice Conversion](https://mlsanity.com/paper/2008.02830/) (2020)
- [ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders](https://mlsanity.com/paper/2004.11012/) (2020)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders](https://mlsanity.com/paper/1912.02613/) (2019)
- [CoMoSVC: Consistency Model-based Singing Voice Conversion](https://mlsanity.com/paper/2401.01792/) (2024)
- [Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech](https://mlsanity.com/paper/2110.04153/) (2021)
- [LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance](https://mlsanity.com/paper/2406.05325/) (2024)

## Built on by (most cited first, 2 of 2)

- [LHQ-SVC: Lightweight and High Quality Singing Voice Conversion Modeling](https://mlsanity.com/paper/2409.08583/) (2024)
- [Vibrato Expression Control for Singing Voice Conversion with Improving Independent Control](https://mlsanity.com/paper/2606.17126/) (2026)

## Abstract (arXiv)

Any-to-any singing voice conversion (SVC) aims to transfer a target singer's timbre to other songs using a short voice sample. However many diffusion model based any-to-any SVC methods, which have achieved impressive results, usually suffered from low efficiency caused by a mass of inference steps. In this paper, we propose LCM-SVC, a latent consistency distillation (LCD) based latent diffusion model (LDM) to accelerate inference speed. We achieved one-step or few-step inference while maintaining the high performance by distilling a pre-trained LDM based SVC model, which had the advantages of timbre decoupling and sound quality. Experimental results show that our proposed method can significantly reduce the inference time and largely preserve the sound quality and timbre similarity comparing with other state-of-the-art SVC models. Audio samples are available at https://sounddemos.github.io/lcm-svc.