mlsanityData to 7 Oct 2026

Synchformer: Efficient Synchronization from Sparse Cues

arXiv 2401.16423 ↗First posted 29 Jan 2024cs.CV cs.LG cs.MMcited by 94

Not on the map. Why?

Abstract (arXiv)

Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

21 of 32 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  12. 2017Objects that Soundcited by 254
9 more
  1. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  2. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  3. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  4. 2021Audio-Visual Synchronisation in the wildcited by 39
  5. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  6. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 18
  7. 2018Dynamic Temporal Alignment of Speech to Lipscited by 13
  8. 2018On Attention Modules for Audio-Visual Synchronizationcited by 13
  9. 2023ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 38; 2026: 50.

94 papers.

Sort:

By area:

  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  3. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  4. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  5. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  6. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  7. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 42
  8. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  9. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  10. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  11. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
  12. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 22
38 more
  1. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  2. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 19
  3. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generationcited by 18
  4. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  5. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  6. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
  7. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decompositioncited by 10
  8. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  9. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  10. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 8
  11. 2026Apollo: Unified Multi-Task Audio-Video Joint Generationcited by 8
  12. 2026MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generationcited by 8
  13. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  14. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  15. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  16. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  17. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalitiescited by 5
  18. 2025Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Editscited by 5
  19. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  20. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  21. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learningcited by 4
  22. 2026Woosh: A Sound Effects Foundation Modelcited by 4
  23. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 4
  24. 2025ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversationcited by 3
  25. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  26. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  27. 2025MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separationcited by 3
  28. 2025Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Videocited by 3
  29. 2025Hear What Matters! Text-conditioned Selective Video-to-Audio Generationcited by 3
  30. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  31. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?cited by 3
  32. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 3
  33. 2026VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generationcited by 3
  34. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  35. 2025SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generationcited by 2
  36. 2025SynchroRaMa : Lip-Synchronized and Emotion-Aware Talking Face Generation via Multi-Modal Emotion Embeddingcited by 2
  37. 2025Video Object Segmentation-Aware Audio Generationcited by 2
  38. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generationcited by 2

50 most cited of 94 All 94 (JSON)