# A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings

arXiv: https://arxiv.org/abs/2410.02239 · HTML: https://mlsanity.com/paper/2410.02239/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling](https://mlsanity.com/paper/2009.02725/) (2020)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining](https://mlsanity.com/paper/1912.06813/) (2019)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [Pretraining Techniques for Sequence-to-Sequence Voice Conversion](https://mlsanity.com/paper/2008.03088/) (2020)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [Non-autoregressive sequence-to-sequence voice conversion](https://mlsanity.com/paper/2104.06793/) (2021)
- [Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion](https://mlsanity.com/paper/2309.02133/) (2023)
- [Automatic Speech Recognition of Non-Native Child Speech for Language Learning Applications](https://mlsanity.com/paper/2306.16710/) (2023)
- [AAS-VC: On the Generalization Ability of Automatic Alignment Search based Non-autoregressive Sequence-to-sequence Voice Conversion](https://mlsanity.com/paper/2309.07598/) (2023)

## Built on by (most cited first, 1 of 1)

- [LinearVC: Linear transformations of self-supervised features through the lens of voice conversion](https://mlsanity.com/paper/2506.01510/) (2025)

## Abstract (arXiv)

Utterances by L2 speakers can be unintelligible due to mispronunciation and improper prosody. In computer-aided language learning systems, textual feedback is often provided using a speech recognition engine. However, an ideal form of feedback for L2 speakers should be so fine-grained that it enables them to detect and diagnose unintelligible parts of L2 speakers' utterances. Inspired by language teachers who correct students' pronunciation through a voice-to-voice process, this pilot study utilizes a unique semi-parallel dataset composed of non-native speakers' (L2) reading aloud, shadowing of native speakers (L1) and their script-shadowing utterances. We explore the technical possibility of replicating the process of an L1 speaker's shadowing L2 speech using Voice Conversion techniques, to create a virtual shadower system. Experimental results demonstrate the feasibility of the VC system in simulating L1's shadowing behavior. The output of the virtual shadower system shows a reasonable similarity to the real L1 shadowing utterances in both linguistic and acoustic aspects.