mlsanityData to 14 Oct 2026

Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

arXiv 2410.19560 ↗First posted 25 Oct 2024cs.CV cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

In recent advancements in unsupervised visual representation learning, the Joint-Embedding Predictive Architecture (JEPA) has emerged as a significant method for extracting visual features from unlabeled imagery through an innovative masking strategy. Despite its success, two primary limitations have been identified: the inefficacy of Exponential Moving Average (EMA) from I-JEPA in preventing entire collapse and the inadequacy of I-JEPA prediction in accurately learning the mean of patch representations. Addressing these challenges, this study introduces a novel framework, namely C-JEPA (Contrastive-JEPA), which integrates the Image-based Joint-Embedding Predictive Architecture with the Variance-Invariance-Covariance Regularization (VICReg) strategy. This integration is designed to effectively learn the variance/covariance for preventing entire collapse and ensuring invariance in the mean of augmented views, thereby overcoming the identified limitations. Through empirical and theoretical evaluations, our work demonstrates that C-JEPA significantly enhances the stability and quality of visual representation learning. When pre-trained on the ImageNet-1K dataset, C-JEPA exhibits rapid and improved convergence in both linear probing and fine-tuning performance metrics.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 54 of its 61 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

54 of 61 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Mask R-CNNcited by 7,277
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 2020Supervised Contrastive Learningcited by 2,829
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2019Contrastive Multiview Codingcited by 1,301
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  13. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  14. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  15. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  16. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  17. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  18. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  19. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  20. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  21. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  22. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  23. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  24. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  25. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  26. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  27. 2021SiT: Self-supervised vIsion Transformercited by 99
  28. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  29. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  30. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  31. 2020Unsupervised Feature Learning by Cross-Level Instance-Group Discriminationcited by 51
  32. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41
  33. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  34. 2022Object-wise Masked Autoencoders for Fast Pre-trainingcited by 15
  35. 2022Bag of Image Patch Embedding Behind the Success of Self-Supervised Learningcited by 10
  36. 2022Siamese Prototypical Contrastive Learningcited by 10
  37. 2022Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlationcited by 9
  38. 2024DMT-JEPA: Discriminative Masked Targets for Joint-Embedding Predictive Architecturecited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 17.

20 papers.

Sort:

By area:

  1. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  2. 2025OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observationcited by 24
  3. 2025Simplifying DINO via Coding Rate Regularizationcited by 21
  4. 2026VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Modelscited by 17
  5. 2025DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecturecited by 4
  6. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 4
  7. 2026BiJEPA: Bi-directional Joint Embedding Predictive Architecture for Symmetric Representation Learningcited by 4
  8. 2026AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modelingcited by 4
  9. 2026Self-Distillation of Hidden Layers for Self-Supervised Representation Learningcited by 3
  10. 2026HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planningcited by 2
  11. 2026The JEPA Paradox in Language: The Geometry of Linguistic Alternativescited by 2
  12. 2026Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architecturescited by 1
8 more
  1. 2026DART: A Vision-Language Foundation Model for Comprehensive Rope Condition Monitoringcited by 0
  2. 2026SegDem: Segmentation helps Demosaicingcited by 0
  3. 2026Three Necessary Principles for Self-Supervised Visual Representation Learningcited by 0
  4. 2026CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representationscited by 0
  5. 2026Mask-Aware Execution for Efficient JEPA Trainingcited by 0
  6. 2026LocalProp: Neuro-Localized Memory-Efficient Backpropagationcited by 0
  7. 2026What masking geometry works best for EEG foundation models?cited by 0
  8. 2026Pythia: Toward Foundation World Models for Multimodal Time Seriescited by 0