mlsanityData to 7 Oct 2026

Perfect match: Improved cross-modal embeddings for audio-visual synchronisation

arXiv 1809.08001 ↗First posted 21 Sep 2018cs.CV cs.SD eess.AScited by 60

Not on the map. Why?

Abstract (arXiv)

This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronization. Here, we set up the problem as one of cross-modal retrieval, where the objective is to find the most relevant audio segment given a short video clip. The method builds on the recent advances in learning representations from cross-modal self-supervision. The main contributions of this paper are as follows: (1) we propose a new learning strategy where the embeddings are learnt via a multi-way matching problem, as opposed to a binary classification (matching or non-matching) problem as proposed by recent papers; (2) we demonstrate that performance of this method far exceeds the existing baselines on the synchronization task; (3) we use the learnt embeddings for visual speech recognition in self-supervision, and show that the performance matches the representations learnt end-to-end in a fully-supervised manner.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 20; 2023: 6; 2024: 9; 2025: 5; 2026: 2.

60 papers.

Sort:

By area:

  1. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  2. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  3. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  4. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  5. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  6. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  7. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  8. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  9. 2021Representation Learning via Global Temporal Alignment and Cycle-Consistencycited by 43
  10. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  11. 2021Audio-Visual Synchronisation in the wildcited by 39
  12. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
38 more
  1. 2024Audio-Synchronized Visual Animationcited by 32
  2. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  3. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  4. 2021UniCon: Unified Context Network for Robust Active Speaker Detectioncited by 27
  5. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  6. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wildcited by 24
  7. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  8. 2021Look Who's Talking: Active Speaker Detection in the Wildcited by 21
  9. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  10. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  11. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 18
  12. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  13. 2021Composable Augmentation Encoding for Video Representation Learningcited by 13
  14. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  15. 2021LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Readingcited by 10
  16. 2021Automated Video Labelling: Identifying Faces by Corroborative Evidencecited by 9
  17. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  18. 2022Lip-to-Speech Synthesis for Arbitrary Speakers in the Wildcited by 8
  19. 2022Towards Effective Image Manipulation Detection with Proposal Contrastive Learningcited by 8
  20. 2023Backchannel Detection and Agreement Estimation from Video with Transformer Networkscited by 7
  21. 2021Contrastive Learning of Global-Local Video Representationscited by 6
  22. 2025UniSync: A Unified Framework for Audio-Visual Synchronizationcited by 6
  23. 2022Speaker Extraction with Co-Speech Gestures Cuecited by 5
  24. 2023WASD: A Wilder Active Speaker Detection Datasetcited by 5
  25. 2022Best of Both Worlds: Multi-task Audio-Visual Automatic Speech Recognition and Active Speaker Detectioncited by 4
  26. 2023ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformerscited by 4
  27. 2023GestSync: Determining who is speaking without a talking headcited by 4
  28. 2021Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusioncited by 3
  29. 2022End-to-End Multi-Person Audio/Visual Automatic Speech Recognitioncited by 3
  30. 2022A Closer Look at Audio-Visual Multi-Person Speech Recognition and Active Speaker Selectioncited by 3
  31. 2024A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detectioncited by 3
  32. 2022End to End Lip Synchronization with a Temporal AutoEncodercited by 2
  33. 2022Multimodal Transformer Distillation for Audio-Visual Synchronizationcited by 2
  34. 2024Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarizationcited by 2
  35. 2025Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigmcited by 2
  36. 2022End-To-End Audiovisual Feature Fusion for Active Speaker Detectioncited by 1
  37. 2022Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priorscited by 1
  38. 2024Pretext Training Algorithms for Event Sequence Datacited by 1

50 most cited of 60 All 60 (JSON)