# Dynamic Temporal Alignment of Speech to Lips

arXiv: https://arxiv.org/abs/1808.06250 · HTML: https://mlsanity.com/paper/1808.06250/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2018.

## Built on by (most cited first, 13 of 13)

- [Synchformer: Efficient Synchronization from Sparse Cues](https://mlsanity.com/paper/2401.16423/) (2024)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers](https://mlsanity.com/paper/2108.11996/) (2021)
- [Neural Dubber: Dubbing for Videos According to Scripts](https://mlsanity.com/paper/2110.08243/) (2021)
- [Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors](https://mlsanity.com/paper/2210.07055/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [GestSync: Determining who is speaking without a talking head](https://mlsanity.com/paper/2310.05304/) (2023)
- [DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention](https://mlsanity.com/paper/2309.06511/) (2023)
- [Let Your Video Listen to Your Music!](https://mlsanity.com/paper/2506.18881/) (2025)
- [End to End Lip Synchronization with a Temporal AutoEncoder](https://mlsanity.com/paper/2203.16224/) (2022)
- [Audio-Visual Fusion Layers for Event Type Aware Video Recognition](https://mlsanity.com/paper/2202.05961/) (2022)
- [Latent Optimal Paths by Gumbel Propagation for Variational Bayesian Dynamic Programming](https://mlsanity.com/paper/2306.02568/) (2023)
- [Conditional Deep Canonical Time Warping](https://mlsanity.com/paper/2412.18234/) (2024)

## Abstract (arXiv)

Many speech segments in movies are re-recorded in a studio during postproduction, to compensate for poor sound quality as recorded on location. Manual alignment of the newly-recorded speech with the original lip movements is a tedious task. We present an audio-to-video alignment method for automating speech to lips alignment, stretching and compressing the audio signal to match the lip movements. This alignment is based on deep audio-visual features, mapping the lips video and the speech signal to a shared representation. Using this shared representation we compute the lip-sync error between every short speech period and every video frame, followed by the determination of the optimal corresponding frame for each short sound period over the entire video clip. We demonstrate successful alignment both quantitatively, using a human perception-inspired metric, as well as qualitatively. The strongest advantage of our audio-to-video approach is in cases where the original voice in unclear, and where a constant shift of the sound can not give a perfect alignment. In these cases state-of-the-art methods will fail.