# Lip2AudSpec: Speech reconstruction from silent lip movements video

arXiv: https://arxiv.org/abs/1710.09798 · HTML: https://mlsanity.com/paper/1710.09798/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2017.

## Built on by (most cited first, 30 of 32)

- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [SVTS: Scalable Video-to-Speech Synthesis](https://mlsanity.com/paper/2205.02058/) (2022)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding](https://mlsanity.com/paper/2308.07787/) (2023)
- [Lip-to-Speech Synthesis in the Wild with Multi-task Learning](https://mlsanity.com/paper/2302.08841/) (2023)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis](https://mlsanity.com/paper/2207.03800/) (2022)
- [Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild](https://mlsanity.com/paper/2209.00642/) (2022)
- [Towards Accurate Lip-to-Speech Synthesis in-the-Wild](https://mlsanity.com/paper/2403.01087/) (2024)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)
- [VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection](https://mlsanity.com/paper/2206.07458/) (2022)
- [3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces](https://mlsanity.com/paper/2104.11532/) (2021)
- [Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations](https://mlsanity.com/paper/2107.12003/) (2021)
- [Learning Speaker-specific Lip-to-Speech Generation](https://mlsanity.com/paper/2206.02050/) (2022)
- [Large-scale unsupervised audio pre-training for video-to-speech synthesis](https://mlsanity.com/paper/2306.15464/) (2023)
- [Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformer](https://mlsanity.com/paper/2309.14586/) (2023)
- [Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend](https://mlsanity.com/paper/2104.14467/) (2021)
- [Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism](https://mlsanity.com/paper/2106.11769/) (2021)
- [Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translator](https://mlsanity.com/paper/2206.02284/) (2022)
- [EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning](https://mlsanity.com/paper/2206.07860/) (2022)
- [Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction](https://mlsanity.com/paper/2207.04213/) (2022)
- [Synthesizing audio from tongue motion during speech using tagged MRI via transformer](https://mlsanity.com/paper/2302.07203/) (2023)
- [On the Audio-visual Synchronization for Lip-to-Speech Synthesis](https://mlsanity.com/paper/2303.00502/) (2023)
- [RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations](https://mlsanity.com/paper/2307.01233/) (2023)
- [Audio-visual video-to-speech synthesis with synthesized input audio](https://mlsanity.com/paper/2307.16584/) (2023)

## Abstract (arXiv)

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.