# Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density

arXiv: https://arxiv.org/abs/2510.05949 · HTML: https://mlsanity.com/paper/2510.05949/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Learning by Reconstruction Produces Uninformative Features For Perception](https://mlsanity.com/paper/2402.11337/) (2024)
- [How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks](https://mlsanity.com/paper/2407.03475/) (2024)
- [Implicit variance regularization in non-contrastive SSL](https://mlsanity.com/paper/2212.04858/) (2022)
- [Minimal Achievable Sufficient Statistic Learning](https://mlsanity.com/paper/1905.07822/) (2019)

## Built on by (most cited first, 13 of 13)

- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [InfoNCE Induces Gaussian Distribution](https://mlsanity.com/paper/2602.24012/) (2026)
- [Pathwise Test-Time Correction for Autoregressive Long Video Generation](https://mlsanity.com/paper/2602.05871/) (2026)
- [A Generalization Theory for JEPA-Based World Models](https://mlsanity.com/paper/2606.27014/) (2026)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [The Universal Normal Embedding](https://mlsanity.com/paper/2603.21786/) (2026)
- [The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models](https://mlsanity.com/paper/2607.13612/) (2026)
- [Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion](https://mlsanity.com/paper/2605.24631/) (2026)
- [Deep Networks Favor Simple Data](https://mlsanity.com/paper/2604.00394/) (2026)
- [Disentanglement Beyond Generative Models with Riemannian ICA](https://mlsanity.com/paper/2605.22531/) (2026)
- [Self-Improving Diffusion Classifiers with Minority Preference Optimization](https://mlsanity.com/paper/2607.03770/) (2026)
- [The JEPA Predictor: A Transferable Operator for Occluded Feature Completion](https://mlsanity.com/paper/2607.16274/) (2026)
- [dRAE: Representation Autoencoder with Hyper-Spherical Codes](https://mlsanity.com/paper/2607.22148/) (2026)

## Abstract (arXiv)

Joint Embedding Predictive Architectures (JEPAs) learn representations able to solve numerous downstream tasks out-of-the-box. JEPAs combine two objectives: (i) a latent-space prediction term, i.e., the representation of a slightly perturbed sample must be predictable from the original sample's representation, and (ii) an anti-collapse term, i.e., not all samples should have the same representation. While (ii) is often considered as an obvious remedy to representation collapse, we uncover that JEPAs' anti-collapse term does much more--it provably estimates the data density. In short, any successfully trained JEPA can be used to get sample probabilities, e.g., for data curation, outlier detection, or simply for density estimation. Our theoretical finding is agnostic of the dataset and architecture used--in any case one can compute the learned probabilities of sample x efficiently and in closed-form using the model's Jacobian matrix at x. Our findings are empirically validated across datasets (synthetic, controlled, and Imagenet) and across different Self Supervised Learning methods falling under the JEPA family (I-JEPA and DINOv2) and on multimodal models, such as MetaCLIP. We denote the method extracting the JEPA learned density as bf JEPA-SCORE.