mlsanityData to 7 Oct 2026

Audio-Visual Synchronisation in the wild

arXiv 2112.04432 ↗First posted 8 Dec 2021cs.CV eess.AScited by 39

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the problem of audio-visual synchronisation applied to videos `in-the-wild' (ie of general classes beyond speech). As a new task, we identify and curate a test set with high audio-visual correlation, namely VGG-Sound Sync. We compare a number of transformer-based architectural variants specifically designed to model audio and visual signals of arbitrary length, while significantly reducing memory requirements during training. We further conduct an in-depth analysis on the curated dataset and define an evaluation metric for open domain audio-visual synchronisation. We apply our method on standard lip reading speech benchmarks, LRS2 and LRS3, with ablations on various aspects. Finally, we set the first benchmark for general audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound Sync video dataset. In all cases, our proposed model outperforms the previous state-of-the-art by a significant margin.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

52 of 69 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  12. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
38 more
  1. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  2. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  3. 2018Deep Audio-Visual Speech Recognitioncited by 318
  4. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  5. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  6. 2018The Sound of Pixelscited by 258
  7. 2017Objects that Soundcited by 254
  8. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  9. 2016Lip Reading Sentences in the Wildcited by 226
  10. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  11. 2021Localizing Visual Sounds the Hard Waycited by 169
  12. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  13. 2019The Sound of Motionscited by 145
  14. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  15. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  16. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  17. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  18. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  19. 2020Music Gesture for Visual Sound Separationcited by 112
  20. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  21. 2017You said that?cited by 99
  22. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  23. 2020Towards Automatic Face-to-Face Translationcited by 95
  24. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  25. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  26. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  27. 20182.5D Visual Soundcited by 63
  28. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  29. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63
  30. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  31. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  32. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  33. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  34. 2020Large-scale multilingual audio visual dubbingcited by 15
  35. 2018Dynamic Temporal Alignment of Speech to Lipscited by 13
  36. 2018On Attention Modules for Audio-Visual Synchronizationcited by 13
  37. 2021Detection of Audio-Video Synchronization Errors Via Event Detectioncited by 6
  38. 2021V-SlowFast Network for Efficient Visual Sound Separationcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 11; 2025: 7; 2026: 6.

39 papers.

Sort:

By area:

  1. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  2. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  3. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  4. 2023Audio-Visual Segmentation with Semanticscited by 73
  5. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  6. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  7. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  8. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  9. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  10. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  11. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 18
  12. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
27 more
  1. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  2. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  3. 2024PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scorescited by 12
  4. 2022Rethinking Audio-visual Synchronization for Active Speaker Detectioncited by 9
  5. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  6. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  7. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  8. 2025UniSync: A Unified Framework for Audio-Visual Synchronizationcited by 6
  9. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  10. 2023Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generationcited by 4
  11. 2023Speech inpainting: Context-based speech synthesis guided by videocited by 4
  12. 2023GestSync: Determining who is speaking without a talking headcited by 4
  13. 2025ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversationcited by 3
  14. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  15. 2022Multimodal Transformer Distillation for Audio-Visual Synchronizationcited by 2
  16. 2023Laughing Matters: Introducing Laughing-Face Generation using Diffusion Modelscited by 2
  17. 2025SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generationcited by 2
  18. 2025Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localizationcited by 2
  19. 2024From Coarse to Fine: Efficient Training for Audio Spectrogram Transformerscited by 1
  20. 2024Interpretable Convolutional SyncNetcited by 1
  21. 2025Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservationcited by 1
  22. 2026MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignmentcited by 1
  23. 2026Enhancing Self-Supervised Talking Head Forgery Detection via a Training-Free Dual-System Frameworkcited by 0
  24. 2026FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precisioncited by 0
  25. 2026AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronizationcited by 0
  26. 2026When Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessmentcited by 0
  27. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0