mlsanityData to 14 Oct 2026

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

arXiv 2610.06226 ↗First posted 5 Oct 2026cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 47 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

43 of 47 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
31 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  7. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  9. 2021AST: Audio Spectrogram Transformercited by 643
  10. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  11. 2017Look, Listen and Learncited by 458
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  13. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  14. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  15. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  16. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  17. 2020Whitening for Self-Supervised Representation Learningcited by 279
  18. 2022Masked Autoencoders that Listencited by 273
  19. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  20. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  21. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  22. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  23. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  24. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  25. 2022MAViL: Masked Audio-Video Learnerscited by 59
  26. 2022Audiovisual Masked Autoencoderscited by 49
  27. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 31
  28. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 16
  29. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  30. 2024EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learningcited by 12
  31. 2026MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learningcited by 2

Built on by

Not cited yet.