# LCANet: End-to-End Lipreading with Cascaded Attention-CTC

arXiv: https://arxiv.org/abs/1803.04988 · HTML: https://mlsanity.com/paper/1803.04988/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2018.

## Built on by (most cited first, 15 of 15)

- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers](https://mlsanity.com/paper/2302.02141/) (2023)
- [Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild](https://mlsanity.com/paper/2209.00642/) (2022)
- [SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory](https://mlsanity.com/paper/2108.13630/) (2021)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Personalized One-Shot Lipreading for an ALS Patient](https://mlsanity.com/paper/2111.01740/) (2021)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation](https://mlsanity.com/paper/2502.05758/) (2025)
- [SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer](https://mlsanity.com/paper/2505.04394/) (2025)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)
- [Learning Speaker-Invariant Visual Features for Lipreading](https://mlsanity.com/paper/2506.07572/) (2025)
- [NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware](https://mlsanity.com/paper/2609.24391/) (2026)

## Abstract (arXiv)

Machine lipreading is a special type of automatic speech recognition (ASR) which transcribes human speech by visually interpreting the movement of related face regions including lips, face, and tongue. Recently, deep neural network based lipreading methods show great potential and have exceeded the accuracy of experienced human lipreaders in some benchmark datasets. However, lipreading is still far from being solved, and existing methods tend to have high error rates on the wild data. In this paper, we propose LCANet, an end-to-end deep neural network based lipreading system. LCANet encodes input video frames using a stacked 3D convolutional neural network (CNN), highway network and bidirectional GRU network. The encoder effectively captures both short-term and long-term spatio-temporal information. More importantly, LCANet incorporates a cascaded attention-CTC decoder to generate output texts. By cascading CTC with attention, it partially eliminates the defect of the conditional independence assumption of CTC within the hidden neural layers, and this yields notably performance improvement as well as faster convergence. The experimental results show the proposed system achieves a 1.3% CER and 3.0% WER on the GRID corpus database, leading to a 12.3% improvement compared to the state-of-the-art methods.