mlsanityData to 19 Oct 2026

Audio-visual speech recognition

283 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: visual, audio-visual, recognition, sign, speech recognition

Key facts. Audio-visual speech recognition: 283 papers on arXiv since 2021 (data through 19 Oct 2026). 118 of its 283 mapped papers (42%) cite “Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction” (2022, arXiv 2201.02184). 3 new papers 2 Oct–19 Oct.

What’s new

2 Oct – 19 Oct 2026

3 new since 2 Oct

  1. VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing
  2. From Sound to Sense: An Anatomy of Semantic Supervision for Speech-Driven Visual Grounding in Remote Sensing Imagery
  3. ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context

Foundations

Most cited by papers here

  1. Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction2022cited by 118 here
  2. End-to-end Audio-visual Speech Recognition with Conformers2021cited by 115 here
  3. Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels2023cited by 84 here
  4. Visual Speech Recognition for Multiple Languages in the Wild2022cited by 55 here
  5. Robust Self-Supervised Audio-Visual Speech Recognition2022cited by 48 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 115 here
  2. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 118 here
  3. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 25 here
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 55 here
  5. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 42 here
  6. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 21 here
  7. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 17 here
  8. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 33 here
  9. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 21 here
15 more it builds on, not in this chain
  1. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 26 here
  2. 2021Sub-word Level Lip Reading With Visual Attentioncited by 42 here
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 21 here
  4. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 48 here
  5. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 14 here
  6. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23 here
  7. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 25 here
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 82 here
  9. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 18 here
  10. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 22 here
  11. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 28 here
  12. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 84 here
  13. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16 here
  14. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 16 here
  15. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35 here

The 24 most cited of 510.

Browse

30 of 283, ranked 6–35

  1. 6Jointly Learning Visual and Auditory Speech Representations from Raw Data202242
  2. 7Sub-word Level Lip Reading With Visual Attention202142
  3. 8Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation202435
  4. 9Large Language Models are Strong Audio-Visual Speech Recognition Learners202433
  5. 10Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring202328
  6. 11Improving Sign Language Translation with Monolingual Data by Sign Back-Translation202126
  7. 12Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video202225
  8. 13VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning202225
  9. 14Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition202223
  10. 15MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation202322
All 30
  1. 16MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens202521
  2. 17AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations202321
  3. 18Audio-Visual Efficient Conformer for Robust Speech Recognition202318
  4. 19BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition202417
  5. 20AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR202316
  6. 21SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision202316
  7. 22Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs202414
  8. 23Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition202214
  9. 24Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production202214
  10. 25Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning202214
  11. 26Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs202512
  12. 27LLMs are Good Sign Language Translators202412
  13. 28Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper202312
  14. 29Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations202511
  15. 30VALLR: Visual ASR Language Model for Lip Reading202511
  16. 31XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception202411
  17. 32Data Augmentation for Sign Language Gloss Translation202111
  18. 33Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach202510
  19. 34SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization202410
  20. 35Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition202410