# On Attention Modules for Audio-Visual Synchronization

arXiv: https://arxiv.org/abs/1812.06071 · HTML: https://mlsanity.com/paper/1812.06071/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 13 of 13)

- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Synchformer: Efficient Synchronization from Sparse Cues](https://mlsanity.com/paper/2401.16423/) (2024)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [Self-supervised object detection from audio-visual correspondence](https://mlsanity.com/paper/2104.06401/) (2021)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors](https://mlsanity.com/paper/2210.07055/) (2022)
- [VGGSounder: Audio-Visual Evaluations for Foundation Models](https://mlsanity.com/paper/2508.08237/) (2025)
- [Detection of Audio-Video Synchronization Errors Via Event Detection](https://mlsanity.com/paper/2104.10116/) (2021)
- [GestSync: Determining who is speaking without a talking head](https://mlsanity.com/paper/2310.05304/) (2023)
- [Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification](https://mlsanity.com/paper/2408.14441/) (2024)
- [Explainable Temporal Attention-based Defect Detection For Fillet Joints in Real-Time Gas Metal Arc Welding Based on Multi-modal Data](https://mlsanity.com/paper/2609.07893/) (2026)

## Abstract (arXiv)

With the development of media and networking technologies, multimedia applications ranging from feature presentation in a cinema setting to video on demand to interactive video conferencing are in great demand. Good synchronization between audio and video modalities is a key factor towards defining the quality of a multimedia presentation. The audio and visual signals of a multimedia presentation are commonly managed by independent workflows - they are often separately authored, processed, stored and even delivered to the playback system. This opens up the possibility of temporal misalignment between the two modalities - such a tendency is often more pronounced in the case of produced content (such as movies). To judge whether audio and video signals of a multimedia presentation are synchronized, we as humans often pay close attention to discriminative spatio-temporal blocks of the video (e.g. synchronizing the lip movement with the utterance of words, or the sound of a bouncing ball at the moment it hits the ground). At the same time, we ignore large portions of the video in which no discriminative sounds exist (e.g. background music playing in a movie). Inspired by this observation, we study leveraging attention modules for automatically detecting audio-visual synchronization. We propose neural network based attention modules, capable of weighting different portions (spatio-temporal blocks) of the video based on their respective discriminative power. Our experiments indicate that incorporating attention modules yields state-of-the-art results for the audio-visual synchronization classification problem.