# EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning

arXiv: https://arxiv.org/abs/2403.09502 · HTML: https://mlsanity.com/paper/2403.09502/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)

## Built on by (most cited first, 12 of 12)

- [CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment](https://mlsanity.com/paper/2505.01237/) (2025)
- [Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization](https://mlsanity.com/paper/2409.07967/) (2024)
- [VGGSounder: Audio-Visual Evaluations for Foundation Models](https://mlsanity.com/paper/2508.08237/) (2025)
- [Disentanglement of Variations with Multimodal Generative Modeling](https://mlsanity.com/paper/2509.23548/) (2025)
- [textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time](https://mlsanity.com/paper/2506.00358/) (2025)
- [MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning](https://mlsanity.com/paper/2606.25225/) (2026)
- [Audio-Visual Continual Test-Time Adaptation without Forgetting](https://mlsanity.com/paper/2602.18528/) (2026)
- [CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition](https://mlsanity.com/paper/2503.23447/) (2025)
- [Improving Audio Event Recognition with Consistency Regularization](https://mlsanity.com/paper/2509.10391/) (2025)
- [Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning](https://mlsanity.com/paper/2604.08147/) (2026)
- [DAVSS: Distilled Audio-Visual State Space Models](https://mlsanity.com/paper/2608.19523/) (2026)
- [LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning](https://mlsanity.com/paper/2610.06226/) (2026)

## Abstract (arXiv)

Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations. However, despite the advantages of data augmentation verified in many learning methods, audio-visual learning has struggled to fully harness these benefits, as augmentations can easily disrupt the correspondence between input pairs. To address this limitation, we introduce EquiAV, a novel framework that leverages equivariance for audio-visual contrastive learning. Our approach begins with extending equivariance to audio-visual learning, facilitated by a shared attention-based transformation predictor. It enables the aggregation of features from diverse augmentations into a representative embedding, providing robust supervision. Notably, this is achieved with minimal computational overhead. Extensive ablation studies and qualitative results verify the effectiveness of our method. EquiAV outperforms previous works across various audio-visual benchmarks. The code is available on https://github.com/JongSuk1/EquiAV.