# Multi-Temporal Lip-Audio Memory for Visual Speech Recognition

arXiv: https://arxiv.org/abs/2305.04542 · HTML: https://mlsanity.com/paper/2305.04542/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 19 of 19)

- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Towards Practical Lipreading with Distilled and Efficient Models](https://mlsanity.com/paper/2007.06504/) (2020)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition](https://mlsanity.com/paper/2003.03206/) (2020)
- [Lip-reading with Densely Connected Temporal Convolutional Networks](https://mlsanity.com/paper/2009.14233/) (2020)
- [Training Strategies for Improved Lip-reading](https://mlsanity.com/paper/2209.01383/) (2022)
- [Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading](https://mlsanity.com/paper/1905.02540/) (2019)
- [Deformation Flow Based Two-Stream Network for Lip Reading](https://mlsanity.com/paper/2003.05709/) (2020)
- [Mutual Information Maximization for Effective Lip Reading](https://mlsanity.com/paper/2003.06439/) (2020)
- [Speaker-adaptive Lip Reading with User-dependent Padding](https://mlsanity.com/paper/2208.04498/) (2022)
- [Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading](https://mlsanity.com/paper/2003.03983/) (2020)
- [Multi-Grained Spatio-temporal Modeling for Lip-reading](https://mlsanity.com/paper/1908.11618/) (2019)

## Built on by (most cited first, 9 of 9)

- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization](https://mlsanity.com/paper/2406.12233/) (2024)
- [Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper](https://mlsanity.com/paper/2309.08535/) (2023)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer](https://mlsanity.com/paper/2505.04394/) (2025)
- [CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge](https://mlsanity.com/paper/2506.02010/) (2025)
- [InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition](https://mlsanity.com/paper/2508.02460/) (2025)

## Abstract (arXiv)

Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements. Some works have been recently presented which use audio signals to supplement visual information. However, existing methods utilize only limited information such as phoneme-level features and soft labels of Automatic Speech Recognition (ASR) networks. In this paper, we present a Multi-Temporal Lip-Audio Memory (MTLAM) that makes the best use of audio signals to complement insufficient information of lip movements. The proposed method is mainly composed of two parts: 1) MTLAM saves multi-temporal audio features produced from short- and long-term audio signals, and the MTLAM memorizes a visual-to-audio mapping to load stored multi-temporal audio features from visual features at the inference phase. 2) We design an audio temporal model to produce multi-temporal audio features capturing the context of neighboring words. In addition, to construct effective visual-to-audio mapping, the audio temporal models can generate audio features time-aligned with visual features. Through extensive experiments, we validate the effectiveness of the MTLAM achieving state-of-the-art performances on two public VSR datasets.