mlsanityData to 14 Oct 2026

ASR is all you need: cross-modal distillation for lip reading

arXiv 1911.12747 ↗First posted 28 Nov 2019cs.CV cs.SD eess.AScited by 88

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data. We achieve this by distilling from an Automatic Speech Recognition (ASR) model that has been trained on a large-scale audio-only corpus. We use a cross-modal distillation method that combines Connectionist Temporal Classification (CTC) with a frame-wise cross-entropy loss. Our contributions are fourfold: (i) we show that ground truth transcriptions are not necessary to train a lip reading system; (ii) we show how arbitrary amounts of unlabelled video data can be leveraged to improve performance; (iii) we demonstrate that distillation significantly speeds up training; and, (iv) we obtain state-of-the-art results on the challenging LRS2 and LRS3 datasets for training only on publicly available data.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 22; 2023: 29; 2024: 10; 2025: 7; 2026: 3.

88 papers.

Sort:

By area:

  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  5. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
  6. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 70
  7. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  8. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  9. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  10. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  11. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  12. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
38 more
  1. 2021Audio-Visual Synchronisation in the wildcited by 40
  2. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  3. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  4. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  5. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  6. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  7. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 31
  8. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  9. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  10. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  11. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  12. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  13. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  14. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 21
  15. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  16. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  17. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  18. 2022Speaker-adaptive Lip Reading with User-dependent Paddingcited by 17
  19. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  20. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  21. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  22. 2023Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognitioncited by 14
  23. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  24. 2024SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronizationcited by 14
  25. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  26. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  27. 2023Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mappingcited by 12
  28. 2021Visual Keyword Spotting with Attentioncited by 11
  29. 2022Deep Multimodal Guidance for Medical Image Classificationcited by 11
  30. 2025VALLR: Visual ASR Language Model for Lip Readingcited by 11
  31. 2021Automated Video Labelling: Identifying Faces by Corroborative Evidencecited by 9
  32. 2023Do VSR Models Generalize Beyond LRS3?cited by 9
  33. 2023OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignmentcited by 8
  34. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  35. 2024Learning Modality Knowledge Alignment for Cross-Modality Transfercited by 8
  36. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  37. 2022Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignmentcited by 7
  38. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7

50 most cited of 88 All 88 (JSON)