# Iteratively Improving Speech Recognition and Voice Conversion

arXiv: https://arxiv.org/abs/2305.15055 · HTML: https://mlsanity.com/paper/2305.15055/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage](https://mlsanity.com/paper/2111.09344/) (2021)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization](https://mlsanity.com/paper/2011.00316/) (2020)
- [VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture](https://mlsanity.com/paper/2006.04154/) (2020)
- [Deep Learning Based Assessment of Synthetic Speech Naturalness](https://mlsanity.com/paper/2104.11673/) (2021)
- [Unsupervised Cross-Domain Singing Voice Conversion](https://mlsanity.com/paper/2008.02830/) (2020)
- [NHSS: A Speech and Singing Parallel Database](https://mlsanity.com/paper/2012.00337/) (2020)
- [Voice Conversion Can Improve ASR in Very Low-Resource Settings](https://mlsanity.com/paper/2111.02674/) (2021)
- [Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition](https://mlsanity.com/paper/2203.15796/) (2022)
- [Hierarchical Diffusion Models for Singing Voice Neural Vocoder](https://mlsanity.com/paper/2210.07508/) (2022)
- [Hierarchical disentangled representation learning for singing voice conversion](https://mlsanity.com/paper/2101.06842/) (2021)
- [Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs Using Dual Domain Adversarial Network & Virtual Domain Pairing](https://mlsanity.com/paper/2302.10536/) (2023)
- [Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding](https://mlsanity.com/paper/2110.04754/) (2021)
- [Improving Voice Separation by Incorporating End-to-end Speech Recognition](https://mlsanity.com/paper/1911.12928/) (2019)
- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [End-to-end lyrics Recognition with Voice to Singing Style Transfer](https://mlsanity.com/paper/2102.08575/) (2021)
- [Non-Parallel Voice Conversion for ASR Augmentation](https://mlsanity.com/paper/2209.06987/) (2022)

## Built on by (most cited first, 2 of 2)

- [LOCKEY: A Novel Approach to Model Authentication and Deepfake Tracking](https://mlsanity.com/paper/2409.07743/) (2024)
- [Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulation](https://mlsanity.com/paper/2512.06304/) (2025)

## Abstract (arXiv)

Many existing works on voice conversion (VC) tasks use automatic speech recognition (ASR) models for ensuring linguistic consistency between source and converted samples. However, for the low-data resource domains, training a high-quality ASR remains to be a challenging task. In this work, we propose a novel iterative way of improving both the ASR and VC models. We first train an ASR model which is used to ensure content preservation while training a VC model. In the next iteration, the VC model is used as a data augmentation method to further fine-tune the ASR model and generalize it to diverse speakers. By iteratively leveraging the improved ASR model to train VC model and vice-versa, we experimentally show improvement in both the models. Our proposed framework outperforms the ASR and one-shot VC baseline models on English singing and Hindi speech domains in subjective and objective evaluations in low-data resource settings.