# UniSync: A Unified Framework for Audio-Visual Synchronization

arXiv: https://arxiv.org/abs/2503.16357 · HTML: https://mlsanity.com/paper/2503.16357/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [Perfect match: Improved cross-modal embeddings for audio-visual synchronisation](https://mlsanity.com/paper/1809.08001/) (2018)
- [What comprises a good talking-head video generation?: A Survey and Benchmark](https://mlsanity.com/paper/2005.03201/) (2020)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices](https://mlsanity.com/paper/2204.02090/) (2022)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Phoneme-to-viseme mappings: the good, the bad, and the ugly](https://mlsanity.com/paper/1805.02934/) (2018)
- [SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing](https://mlsanity.com/paper/2409.03605/) (2024)
- [A Novel Speech-Driven Lip-Sync Model with CNN and LSTM](https://mlsanity.com/paper/2205.00916/) (2022)

## Built on by (most cited first, 6 of 6)

- [Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning](https://mlsanity.com/paper/2504.18810/) (2025)
- [OnlineHOI: Towards Online Human-Object Interaction Generation and Perception](https://mlsanity.com/paper/2509.12250/) (2025)
- [AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities](https://mlsanity.com/paper/2607.24821/) (2026)
- [ASemConsist: Adaptive Semantic Feature Control for Training-Free Identity-Consistent Generation](https://mlsanity.com/paper/2512.23245/) (2025)
- [E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis](https://mlsanity.com/paper/2601.07877/) (2026)
- [SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning](https://mlsanity.com/paper/2608.16220/) (2026)

## Abstract (arXiv)

Precise audio-visual synchronization in speech videos is crucial for content quality and viewer comprehension. Existing methods have made significant strides in addressing this challenge through rule-based approaches and end-to-end learning techniques. However, these methods often rely on limited audio-visual representations and suboptimal learning strategies, potentially constraining their effectiveness in more complex scenarios. To address these limitations, we present UniSync, a novel approach for evaluating audio-visual synchronization using embedding similarities. UniSync offers broad compatibility with various audio representations (e.g., Mel spectrograms, HuBERT) and visual representations (e.g., RGB images, face parsing maps, facial landmarks, 3DMM), effectively handling their significant dimensional differences. We enhance the contrastive learning framework with a margin-based loss component and cross-speaker unsynchronized pairs, improving discriminative capabilities. UniSync outperforms existing methods on standard datasets and demonstrates versatility across diverse audio-visual representations. Its integration into talking face generation frameworks enhances synchronization quality in both natural and AI-generated content.