# VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices

arXiv: https://arxiv.org/abs/2204.02090 · HTML: https://mlsanity.com/paper/2204.02090/

## Key facts

- Cited by 18 papers in the map (data through 7 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Perfect match: Improved cross-modal embeddings for audio-visual synchronisation](https://mlsanity.com/paper/1809.08001/) (2018)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representations](https://mlsanity.com/paper/2104.08506/) (2021)
- [Selective Listening by Synchronizing Speech with Lips](https://mlsanity.com/paper/2106.07150/) (2021)
- [VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer](https://mlsanity.com/paper/2203.04099/) (2022)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [End to End Lip Synchronization with a Temporal AutoEncoder](https://mlsanity.com/paper/2203.16224/) (2022)

## Built on by (most cited first, 18 of 18)

- [Synchformer: Efficient Synchronization from Sparse Cues](https://mlsanity.com/paper/2401.16423/) (2024)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors](https://mlsanity.com/paper/2210.07055/) (2022)
- [SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis](https://mlsanity.com/paper/2310.15247/) (2023)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Audio-driven Talking Face Generation with Stabilized Synchronization Loss](https://mlsanity.com/paper/2307.09368/) (2023)
- [UniSync: A Unified Framework for Audio-Visual Synchronization](https://mlsanity.com/paper/2503.16357/) (2025)
- [ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformers](https://mlsanity.com/paper/2303.11551/) (2023)
- [Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation](https://mlsanity.com/paper/2305.19556/) (2023)
- [Speech inpainting: Context-based speech synthesis guided by video](https://mlsanity.com/paper/2306.00489/) (2023)
- [GestSync: Determining who is speaking without a talking head](https://mlsanity.com/paper/2310.05304/) (2023)
- [Multimodal Transformer Distillation for Audio-Visual Synchronization](https://mlsanity.com/paper/2210.15563/) (2022)
- [Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models](https://mlsanity.com/paper/2305.08854/) (2023)
- [Interpretable Convolutional SyncNet](https://mlsanity.com/paper/2409.00971/) (2024)
- [Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation](https://mlsanity.com/paper/2507.20953/) (2025)
- [SyncLipMAE: Contrastive Masked Pretraining for Audio-Visual Talking-Face Representation](https://mlsanity.com/paper/2510.10069/) (2025)
- [LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild](https://mlsanity.com/paper/2602.00189/) (2026)
- [Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection](https://mlsanity.com/paper/2606.15117/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audio-visual correspondence score. We propose an audio-visual cross-modal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/