mlsanityData to 14 Oct 2026

End-to-end Audio-visual Speech Recognition with Conformers

arXiv 2102.06657 ↗First posted 12 Feb 2021cs.CV eess.AScited by 151

Not on the map. Why?

Abstract (arXiv)

In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

27 of 37 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2016Language Modeling with Gated Convolutional Networkscited by 702
  8. 2018Deep Audio-Visual Speech Recognitioncited by 322
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  10. 2016Lip Reading Sentences in the Wildcited by 228
  11. 2020Lipreading using Temporal Convolutional Networkscited by 102
  12. 2019Language Modeling with Deep Transformerscited by 96
15 more
  1. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  2. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 80
  3. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  4. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  5. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  6. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  7. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  8. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  9. 2018Large-Scale Visual Speech Recognitioncited by 57
  10. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 45
  11. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 41
  12. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  13. 2018End-to-End Speech Recognition From the Raw Waveformcited by 10
  14. 2020How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognitioncited by 8
  15. 2019Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 29; 2023: 41; 2024: 28; 2025: 35; 2026: 13.

151 papers.

Sort:

By area:

  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  2. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  4. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  5. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  6. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  7. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  8. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  9. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  10. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  11. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  12. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
38 more
  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  2. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 41
  3. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  4. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  5. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  6. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  7. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  8. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  9. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 24
  10. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 22
  11. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  12. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  13. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  14. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 21
  15. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  16. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 19
  17. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  18. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  19. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  20. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  21. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  22. 2023Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognitioncited by 14
  23. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  24. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  25. 2024SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronizationcited by 14
  26. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 13
  27. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  28. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  29. 2023Audio-Visual Speech Enhancement with Score-Based Generative Modelscited by 12
  30. 2023Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mappingcited by 12
  31. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  32. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  33. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  34. 2025VALLR: Visual ASR Language Model for Lip Readingcited by 11
  35. 2022CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognitioncited by 10
  36. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  37. 2022Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillationcited by 9
  38. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9

50 most cited of 151 All 151 (JSON)