mlsanityData to 14 Oct 2026

AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model

arXiv 2308.07593 ↗First posted 15 Aug 2023cs.CV cs.MM eess.AScited by 16

Not on the map. Why?

Abstract (arXiv)

Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this paper, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used LRS3 dataset.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 66 of its 97 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

66 of 97 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2016Layer Normalizationcited by 5,669
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  11. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
38 more
  1. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  2. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  4. 2018Born Again Neural Networkscited by 546
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2014Memory Networkscited by 363
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2018Deep Audio-Visual Speech Recognitioncited by 322
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  11. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  12. 2016Lip Reading Sentences in the Wildcited by 228
  13. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  14. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  15. 2021Direct speech-to-speech translation with discrete unitscited by 149
  16. 2020Lipreading using Temporal Convolutional Networkscited by 102
  17. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  18. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  19. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  20. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 80
  21. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  22. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  23. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  24. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  25. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  26. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  27. 2018Large-Scale Visual Speech Recognitioncited by 57
  28. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  29. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 45
  30. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  31. 2020DeVLBert: Learning Deconfounded Visio-Linguistic Representationscited by 42
  32. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 41
  33. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  34. 2017Lip2AudSpec: Speech reconstruction from silent lip movements videocited by 32
  35. 2021Self-Supervised Visual Representations Learning by Contrastive Mask Predictioncited by 31
  36. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  37. 2020Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognitioncited by 30
  38. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 7; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  2. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  3. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  4. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  5. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  6. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  7. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  8. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 7
  9. 2024Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Languagecited by 5
  10. 2025Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learningcited by 5
  11. 2024AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognitioncited by 4
  12. 2025From Waveforms to Pixels: A Survey on Audio-Visual Segmentationcited by 3
4 more
  1. 2024Enhancing CTC-Based Visual Speech Recognitioncited by 0
  2. 2025Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusioncited by 0
  3. 2025ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extractioncited by 0
  4. 2026Diffusion Large Language Models for Visual Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learners
  2. 2024SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization
  3. 2026OCR-Enhanced Multimodal ASR Can Read While Listening
  4. 2025Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
  5. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
  6. 2024Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimization
  7. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
  8. 2024JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition