mlsanityData to 14 Oct 2026

Self-Supervised Learning of Audio-Visual Objects from Video

arXiv 2008.04237 ↗First posted 10 Aug 2020cs.CV cs.SD eess.AScited by 188

Not on the map. Why?

Abstract (arXiv)

Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks: (a) multi-speaker sound source separation, (b) localizing and tracking speakers, (c) correcting misaligned audio-visual data, and (d) active speaker detection. Using our representation, these tasks can be solved entirely by training on unlabeled video, without the aid of object detectors. We also demonstrate the generality of our method by applying it to non-human speakers, including cartoons and puppets.Our model significantly outperforms other self-supervised approaches, and obtains performance competitive with methods that use supervised face detection.

Key facts. Cited by 188 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 53; 2023: 38; 2024: 27; 2025: 20; 2026: 9.

188 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  3. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  4. 2021Concealed Object Detectioncited by 212
  5. 2021Localizing Visual Sounds the Hard Waycited by 170
  6. 2022Audio-Visual Segmentationcited by 151
  7. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  8. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  9. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  10. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 97
  11. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  12. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
38 more
  1. 2022Localizing Visual Sounds the Easy Waycited by 82
  2. 2023Audio-Visual Segmentation with Semanticscited by 75
  3. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  4. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  5. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  6. 2022Audio Self-supervised Learning: A Surveycited by 65
  7. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  8. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  9. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  10. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 53
  11. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  12. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  13. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  14. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 47
  15. 2022Visual Acoustic Matchingcited by 45
  16. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  17. 2021Time-Equivariant Contrastive Video Representation Learningcited by 44
  18. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 44
  19. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  20. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  21. 2021Audio-Visual Synchronisation in the wildcited by 40
  22. 2023Egocentric Audio-Visual Object Localizationcited by 40
  23. 2021MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsingcited by 39
  24. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 36
  25. 2023Connecting Multi-modal Contrastive Representationscited by 36
  26. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  27. 2021Cross-modal Consensus Network for Weakly Supervised Temporal Action Localizationcited by 34
  28. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 34
  29. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  30. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 33
  31. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  32. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  33. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  34. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 31
  35. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  36. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  37. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  38. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28

50 most cited of 188 All 188 (JSON)