# Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

arXiv: https://arxiv.org/abs/2207.05692 · HTML: https://mlsanity.com/paper/2207.05692/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 5 Jun 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 24 of 24)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [ASR is all you need: cross-modal distillation for lip reading](https://mlsanity.com/paper/1911.12747/) (2019)
- [Relation Distillation Networks for Video Object Detection](https://mlsanity.com/paper/1908.09511/) (2019)
- [Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis](https://mlsanity.com/paper/2005.08209/) (2020)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Towards Practical Lipreading with Distilled and Efficient Models](https://mlsanity.com/paper/2007.06504/) (2020)
- [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019)
- [Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition](https://mlsanity.com/paper/2003.03206/) (2020)
- [Deep Lip Reading: a comparison of models and an online application](https://mlsanity.com/paper/1806.06053/) (2018)
- [Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading](https://mlsanity.com/paper/1905.02540/) (2019)
- [Deformation Flow Based Two-Stream Network for Lip Reading](https://mlsanity.com/paper/2003.05709/) (2020)
- [Learn an Effective Lip Reading Model without Pains](https://mlsanity.com/paper/2011.07557/) (2020)
- [Mutual Information Maximization for Effective Lip Reading](https://mlsanity.com/paper/2003.06439/) (2020)
- [Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading](https://mlsanity.com/paper/2003.03983/) (2020)
- [Multi-Grained Spatio-temporal Modeling for Lip-reading](https://mlsanity.com/paper/1908.11618/) (2019)
- [3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition](https://mlsanity.com/paper/1706.05739/) (2017)
- [Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention](https://mlsanity.com/paper/2012.14360/) (2020)

## Built on by (most cited first, 1 of 1)

- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)

## Abstract (arXiv)

In this work, we propose a technique to transfer speech recognition capabilities from audio speech recognition systems to visual speech recognizers, where our goal is to utilize audio data during lipreading model training. Impressive progress in the domain of speech recognition has been exhibited by audio and audio-visual systems. Nevertheless, there is still much to be explored with regards to visual speech recognition systems due to the visual ambiguity of some phonemes. To this end, the development of visual speech recognition models is crucial given the instability of audio models. The main contributions of this work are i) building on recent state-of-the-art word-based lipreading models by integrating sequence-level and frame-level Knowledge Distillation (KD) to their systems; ii) leveraging audio data during training visual models, a feat which has not been utilized in prior word-based work; iii) proposing the Gaussian-shaped averaging in frame-level KD, as an efficient technique that aids the model in distilling knowledge at the sequence model encoder. This work proposes a novel and competitive architecture for lip-reading, as we demonstrate a noticeable improvement in performance, setting a new benchmark equals to 88.64% on the LRW dataset.