# Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning

arXiv: https://arxiv.org/abs/2505.12477 · HTML: https://mlsanity.com/paper/2505.12477/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 72 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology](https://mlsanity.com/paper/2408.00738/) (2024)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)

## Built on by (most cited first, 30 of 36)

- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [AI Must Embrace Specialization via Superhuman Adaptable Intelligence](https://mlsanity.com/paper/2602.23643/) (2026)
- [Why and How Auxiliary Tasks Improve JEPA Representations](https://mlsanity.com/paper/2509.12249/) (2025)
- [Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction](https://mlsanity.com/paper/2603.16281/) (2026)
- [FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed](https://mlsanity.com/paper/2507.03779/) (2025)
- [WIND: Weather Inverse Diffusion for Zero-Shot Atmospheric Modeling](https://mlsanity.com/paper/2602.03924/) (2026)
- [JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics](https://mlsanity.com/paper/2606.14813/) (2026)
- [Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency](https://mlsanity.com/paper/2608.12939/) (2026)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Self-Supervised Learning as Discrete Communication](https://mlsanity.com/paper/2602.09764/) (2026)
- [Gaussian Joint Embeddings For Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.26799/) (2026)
- [Learn from your own latents and not from tokens: A sample-complexity theory](https://mlsanity.com/paper/2605.27734/) (2026)
- [Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning](https://mlsanity.com/paper/2606.07770/) (2026)
- [Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules](https://mlsanity.com/paper/2608.22642/) (2026)
- [Learning Sparse Visual Representations via Spatial-Semantic Factorization](https://mlsanity.com/paper/2602.01905/) (2026)
- [Mutual information and task-relevant latent dimensionality](https://mlsanity.com/paper/2602.08105/) (2026)
- [Mixed Magnification Aggregation for Generalizable Region-Level Representations in Computational Pathology](https://mlsanity.com/paper/2602.22176/) (2026)
- [COP-GEN: Latent Diffusion Transformer for Copernicus Earth Observation Data](https://mlsanity.com/paper/2603.03239/) (2026)
- [IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.15263/) (2026)
- [A Machine Learning Framework for Turbofan Health Estimation via Inverse Problem Formulation](https://mlsanity.com/paper/2604.08460/) (2026)
- [Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data](https://mlsanity.com/paper/2604.24662/) (2026)
- [Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation](https://mlsanity.com/paper/2606.00514/) (2026)
- [The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail](https://mlsanity.com/paper/2606.04010/) (2026)
- [Group Contrastive Learning for Weakly Paired Multimodal Data](https://mlsanity.com/paper/2602.04021/) (2026)
- [ToLL: Topological Layout Learning with Asymmetric Cross-View Structural Distillation for 3D Scene Graph Generation Pretraining](https://mlsanity.com/paper/2603.28178/) (2026)
- [Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance](https://mlsanity.com/paper/2604.16086/) (2026)
- [Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation](https://mlsanity.com/paper/2605.10290/) (2026)
- [Mask-supervised Object-centric Representation Learning with LeJEPA](https://mlsanity.com/paper/2607.02404/) (2026)
- [CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding](https://mlsanity.com/paper/2607.09543/) (2026)

## Abstract (arXiv)

Reconstruction and joint embedding have emerged as two leading paradigms in Self Supervised Learning (SSL). Reconstruction methods focus on recovering the original sample from a different view in input space. On the other hand, joint embedding methods align the representations of different views in latent space. Both approaches offer compelling advantages, yet practitioners lack clear guidelines for choosing between them. In this work, we unveil the core mechanisms that distinguish each paradigm. By leveraging closed form solutions for both approaches, we precisely characterize how the view generation process, e.g. data augmentation, impacts the learned representations. We then demonstrate that, unlike supervised learning, both SSL paradigms require a minimal alignment between augmentations and irrelevant features to achieve asymptotic optimality with increasing sample size. Our findings indicate that in scenarios where these irrelevant features have a large magnitude, joint embedding methods are preferable because they impose a strictly weaker alignment condition compared to reconstruction based methods. These results not only clarify the trade offs between the two paradigms but also substantiate the empirical success of joint embedding approaches on real world challenging datasets.