mlsanityData to 19 Oct 2026

Simulating Native Speaker Shadowing for Nonnative Speech Assessment with Latent Speech Representations

arXiv 2409.11742 ↗First posted 18 Sep 2024cs.SD eess.AScited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Evaluating speech intelligibility is a critical task in computer-aided language learning systems. Traditional methods often rely on word error rates (WER) provided by automatic speech recognition (ASR) as intelligibility scores. However, this approach has significant limitations due to notable differences between human speech recognition (HSR) and ASR. A promising alternative is to involve a native (L1) speaker in shadowing what nonnative (L2) speakers say. Breakdowns or mispronunciations in the L1 speaker's shadowing utterance can serve as indicators for assessing L2 speech intelligibility. In this study, we propose a speech generation system that simulates the L1 shadowing process using voice conversion (VC) techniques and latent speech representations. Our experimental results demonstrate that this method effectively replicates the L1 shadowing process, offering an innovative tool to evaluate L2 speech intelligibility. Notably, systems that utilize self-supervised speech representations (S3R) show a higher degree of similarity to real L1 shadowing utterances in both linguistic accuracy and naturalness.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 19 of its 31 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

19 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Non-autoregressive sequence-to-sequence voice conversion14 Apr

Other references · 18

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  5. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  6. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 153
  7. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 68
  8. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 53
  9. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 49
  10. 2019Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretrainingcited by 41
  11. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 26
  12. 2020FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulationcited by 24
6 more
  1. 2022Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Unitscited by 18
  2. 2020Pretraining Techniques for Sequence-to-Sequence Voice Conversioncited by 17
  3. 2022A Comparative Study of Self-supervised Speech Representation Based Voice Conversioncited by 12
  4. 2024UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalizationcited by 11
  5. 2023Evaluating Methods for Ground-Truth-Free Foreign Accent Conversioncited by 6
  6. 2023AAS-VC: On the Generalization Ability of Automatic Alignment Search based Non-autoregressive Sequence-to-sequence Voice Conversioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings
  2. 2025A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Rater's Shadowing and Sequence-to-sequence Voice Conversion
  3. 2026Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems
  4. 2025Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data
  5. 2021Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning
  6. 2023Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
  7. 2024Disentangling segmental and prosodic factors to non-native speech comprehensibility
  8. 2024Towards Improving NAM-to-Speech Synthesis Intelligibility using Self-Supervised Speech Models