# Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading

arXiv: https://arxiv.org/abs/2003.03983 · HTML: https://mlsanity.com/paper/2003.03983/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 12 of 12)

- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [Multi-Temporal Lip-Audio Memory for Visual Speech Recognition](https://mlsanity.com/paper/2305.04542/) (2023)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading](https://mlsanity.com/paper/2404.11979/) (2024)
- [Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading](https://mlsanity.com/paper/2108.03543/) (2021)
- [Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models](https://mlsanity.com/paper/2207.05692/) (2022)
- [RAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views](https://mlsanity.com/paper/2409.05307/) (2024)

## Abstract (arXiv)

Lip-reading aims to infer the speech content from the lip movement sequence and can be seen as a typical sequence-to-sequence (seq2seq) problem which translates the input image sequence of lip movements to the text sequence of the speech content. However, the traditional learning process of seq2seq models always suffers from two problems: the exposure bias resulted from the strategy of "teacher-forcing", and the inconsistency between the discriminative optimization target (usually the cross-entropy loss) and the final evaluation metric (usually the character/word error rate). In this paper, we propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems. On the one hand, we introduce the evaluation metric (refers to the character error rate in this paper) as a form of reward to optimize the model together with the original discriminative target. On the other hand, inspired by the local perception property of convolutional operation, we perform a pseudo-convolutional operation on the reward and loss dimension, so as to take more context around each time step into account to generate a robust reward and loss for the whole optimization. Finally, we perform a thorough comparison and evaluation on both the word-level and sentence-level benchmarks. The results show a significant improvement over other related methods, and report either a new state-of-the-art performance or a competitive accuracy on all these challenging benchmarks, which clearly proves the advantages of our approach.