Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning
Not on the map. Why?
Abstract (arXiv)
Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) for representation learning using the Masked Image Modeling framework. IJEPA drives representations to capture useful semantic information by predicting in latent rather than input space. However, IJEPA relies on carefully designed context and target windows to avoid representational collapse. The encoder modules in IJEPA cannot adaptively modulate the type of predicted and/or target features based on the feasibility of the masked prediction task as they are not given sufficient information of both context and targets. Based on the intuition that in natural images, information has a strong spatial bias with spatially local regions being highly predictive of one another compared to distant ones. We condition the target encoder and context encoder modules in IJEPA with positions of context and target windows respectively. Our "conditional" encoders show performance gains on several image classification benchmark datasets, improved robustness to context window size and sample-efficiency during pretraining.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 14 Oct 2024.
Builds on
17 of 23 references.
References · 17
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
- 2017Large Batch Training of Convolutional Networkscited by 535
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
- 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
5 more
- 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
- 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
- 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
- 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
- 2023Stochastic positional embeddings improve masked image modelingcited by 12
Built on by
Citing papers per year: 2026: 1.
1 paper.