mlsanityData to 14 Oct 2026

Curriculum-Based Noise Adaptation for Phoneme-to-Text Reconstruction in Visual Speech Recognition

arXiv 2609.20839 ↗First posted 30 Jul 2026cs.CL cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Phoneme-centric visual speech recognition reconstructs sentences from intermediate phoneme predictions, making overall recognition performance highly dependent on the robustness of the phoneme-to-text reconstruction model. Existing reconstruction approaches are commonly trained on clean phoneme sequences or synthetically corrupted inputs, leading to a mismatch between training conditions and the realistic phoneme prediction errors encountered during inference. To address this limitation, this paper proposes progressive error curriculum training (PECT). This curriculum learning framework progressively adapts a No Language Left Behind (NLLB)-based phoneme-to-text reconstruction model using synthetic phoneme perturbations, multi-domain pseudo-labels, and target-domain pseudo-labels generated by a visual speech recognizer. By gradually exposing the reconstruction model to increasingly realistic phoneme prediction errors, the proposed framework improves robustness while preserving sentence-reconstruction accuracy. Experiments on the LRS2 and LRS3 benchmarks demonstrate that PECT consistently improves reconstruction performance across multiple phoneme-based visual speech recognition frontends, including visual automatic speech recognition (V-ASR), point visual automatic speech recognition (PV-ASR), and head-pose-aware visual speech recognition (HP-VSR) variants. In particular, PECT reduces the word error rate (WER) of HP-VSR-FiLMFuse (L4) from 23.3% to 22.2% on LRS2 and reduces the WER of HP-VSR-ResFiLM from 30.3% to 29.7% on LRS3. Comprehensive ablation studies and qualitative analyses further demonstrate the effectiveness of progressively adapting the reconstruction model to realistic phoneme prediction errors. These results show that PECT provides an effective and generalizable curriculum learning strategy for phoneme-to-text reconstruction in phoneme-centric visual speech recognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

33 of 42 references.

References · 33

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  7. 2015Listen, Attend and Spellcited by 543
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  9. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  10. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 316
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
21 more
  1. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  2. 2021Curriculum Learning: A Surveycited by 259
  3. 2017Automated Curriculum Learning for Neural Networkscited by 257
  4. 2016Lip Reading Sentences in the Wildcited by 228
  5. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  6. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  7. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  8. 2018Large-Scale Visual Speech Recognitioncited by 57
  9. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  10. 2020Data Augmentation for Training Dialog Models Robust to Speech Recognition Errorscited by 17
  11. 2022SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguationcited by 14
  12. 2025VALLR: Visual ASR Language Model for Lip Readingcited by 11
  13. 2023A Chat About Boring Problems: Studying GPT-based text normalizationcited by 10
  14. 2021An Approach to Improve Robustness of NLP Systems against ASR Errorscited by 8
  15. 2024MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialoguescited by 8
  16. 2018Phoneme-to-viseme mappings: the good, the bad, and the uglycited by 7
  17. 2025Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstructioncited by 7
  18. 2022InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASRcited by 5
  19. 2023Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translationcited by 5
  20. 2022Filter and evolve: progressive pseudo label refining for semi-supervised automatic speech recognitioncited by 4
  21. 2024Curriculum Learning for Cross-Lingual Data-to-Text Generation With Noisy Datacited by 1

Built on by

Not cited yet.