mlsanityData to 7 Oct 2026

EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning

arXiv 2403.09502 ↗First posted 14 Mar 2024cs.LG cs.AI cs.MMcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations. However, despite the advantages of data augmentation verified in many learning methods, audio-visual learning has struggled to fully harness these benefits, as augmentations can easily disrupt the correspondence between input pairs. To address this limitation, we introduce EquiAV, a novel framework that leverages equivariance for audio-visual contrastive learning. Our approach begins with extending equivariance to audio-visual learning, facilitated by a shared attention-based transformation predictor. It enables the aggregation of features from diverse augmentations into a representative embedding, providing robust supervision. Notably, this is achieved with minimal computational overhead. Extensive ablation studies and qualitative results verify the effectiveness of our method. EquiAV outperforms previous works across various audio-visual benchmarks. The code is available on https://github.com/JongSuk1/EquiAV.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
28 more
  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  6. 2017Look, Listen and Learncited by 452
  7. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  8. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  9. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  12. 2022Masked Autoencoders that Listencited by 271
  13. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  14. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  15. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  16. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  17. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  18. 2021Robust Audio-Visual Instance Discriminationcited by 80
  19. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  20. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  21. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
  22. 2020On Compositions of Transformations in Contrastive Self-Supervised Learningcited by 34
  23. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 31
  24. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 30
  25. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27
  26. 2022EquiMod: An Equivariance Module to Improve Self-Supervised Learningcited by 20
  27. 2023Audio-Visual Contrastive Learning with Temporal Self-Supervisioncited by 18
  28. 2021Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 3.

10 papers.

  1. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
  2. 2024Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localizationcited by 10
  3. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  4. 2025Disentanglement of Variations with Multimodal Generative Modelingcited by 4
  5. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  6. 2026Audio-Visual Continual Test-Time Adaptation without Forgettingcited by 1
  7. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  8. 2025Improving Audio Event Recognition with Consistency Regularizationcited by 0
  9. 2026Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learningcited by 0
  10. 2026MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learningcited by 0