# A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading

arXiv: https://arxiv.org/abs/1908.04917 · HTML: https://mlsanity.com/paper/1908.04917/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 25 of 25)

- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation](https://mlsanity.com/paper/2303.00628/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition](https://mlsanity.com/paper/2303.05309/) (2023)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers](https://mlsanity.com/paper/2302.02141/) (2023)
- [Learning Cross-lingual Visual Speech Representations](https://mlsanity.com/paper/2303.09455/) (2023)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides](https://mlsanity.com/paper/2504.15066/) (2025)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset](https://mlsanity.com/paper/2301.10180/) (2023)
- [Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder](https://mlsanity.com/paper/2308.08488/) (2023)
- [A Large-scale Universal Evaluation Benchmark For Face Forgery Detection](https://mlsanity.com/paper/2406.09181/) (2024)
- [AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies](https://mlsanity.com/paper/2402.13152/) (2024)
- [MCF-VC: Mitigate Catastrophic Forgetting in Class-Incremental Learning for Multimodal Video Captioning](https://mlsanity.com/paper/2402.17680/) (2024)
- [ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.04635/) (2025)
- [FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling](https://mlsanity.com/paper/2509.12052/) (2025)
- [Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks](https://mlsanity.com/paper/2110.06507/) (2021)
- [Automated Speaker Independent Visual Speech Recognition: A Comprehensive Survey](https://mlsanity.com/paper/2306.08314/) (2023)
- [JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition](https://mlsanity.com/paper/2403.18843/) (2024)
- [RAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views](https://mlsanity.com/paper/2409.05307/) (2024)
- [STSA: Spatial-Temporal Semantic Alignment for Visual Dubbing](https://mlsanity.com/paper/2503.23039/) (2025)
- [Learning Speaker-Invariant Visual Features for Lipreading](https://mlsanity.com/paper/2506.07572/) (2025)
- [VALLR-Pin: Uncertainty-Factorized Visual Speech Recognition for Mandarin with Pinyin Guidance](https://mlsanity.com/paper/2512.20032/) (2025)
- [Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment](https://mlsanity.com/paper/2603.21808/) (2026)

## Abstract (arXiv)

Lip reading aims at decoding texts from the movement of a speaker's mouth. In recent years, lip reading methods have made great progress for English, at both word-level and sentence-level. Unlike English, however, Chinese Mandarin is a tone-based language and relies on pitches to distinguish lexical or grammatical meaning, which significantly increases the ambiguity for the lip reading task. In this paper, we propose a Cascade Sequence-to-Sequence Model for Chinese Mandarin (CSSMCM) lip reading, which explicitly models tones when predicting sentence. Tones are modeled based on visual information and syntactic structure, and are used to predict sentence along with visual information and syntactic structure. In order to evaluate CSSMCM, a dataset called CMLR (Chinese Mandarin Lip Reading) is collected and released, consisting of over 100,000 natural sentences from China Network Television website. When trained on CMLR dataset, the proposed CSSMCM surpasses the performance of state-of-the-art lip reading frameworks, which confirms the effectiveness of explicit modeling of tones for Chinese Mandarin lip reading.