# Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations

arXiv: https://arxiv.org/abs/2602.01456 · HTML: https://mlsanity.com/paper/2602.01456/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 72 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [On the duality between contrastive and non-contrastive self-supervised learning](https://mlsanity.com/paper/2206.02574/) (2022)
- [Statistical and Topological Properties of Sliced Probability Divergences](https://mlsanity.com/paper/2003.05783/) (2020)
- [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2303.03307/) (2023)
- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations](https://mlsanity.com/paper/2209.14905/) (2022)
- [Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization](https://mlsanity.com/paper/2411.15931/) (2024)

## Built on by (most cited first, 10 of 10)

- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [VISReg: Variance-Invariance-Sketching Regularization for JEPA training](https://mlsanity.com/paper/2606.02572/) (2026)
- [LpWM: A Case for Sparse Representations in World Models](https://mlsanity.com/paper/2608.22764/) (2026)
- [ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives](https://mlsanity.com/paper/2609.01041/) (2026)
- [Three Necessary Principles for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2608.08309/) (2026)
- [Capturing Cardiac Cyclicity through Phase-Equivariant Self-Supervised Learning](https://mlsanity.com/paper/2608.21147/) (2026)
- [AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving](https://mlsanity.com/paper/2609.34085/) (2026)
- [Anisotropic Representations Improve Planning in JEPA World Models](https://mlsanity.com/paper/2609.37441/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)
- [DSReg: Provably Recovering Individual World Latents without Reconstruction](https://mlsanity.com/paper/2610.09457/) (2026)

## Abstract (arXiv)

Joint-Embedding Predictive Architectures (JEPA) learn view-invariant representations and admit projection-based distribution matching for collapse prevention. Existing approaches regularize representations towards isotropic Gaussian distributions, but inherently favor dense representations and fail to capture the key property of sparsity observed in efficient representations. We introduce Rectified Distribution Matching Regularization (RDMReg), a sliced two-sample distribution-matching loss that aligns representations to a Rectified Generalized Gaussian (RGG) distribution. RGG enables explicit control over expected ℓ0 norm through rectification, while its continuous truncated component admits a maximum-entropy characterization under expected ℓp norm and support constraints. Equipping JEPAs with RDMReg yields Rectified LpJEPA, which strictly generalizes prior Gaussian-based JEPAs. Empirically, Rectified LpJEPA learns sparse, non-negative representations with favorable sparsity--performance trade-offs and competitive downstream performance on image classification benchmarks, showing that RDMReg can enforce sparsity while preserving task-relevant information.