# LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures

arXiv: https://arxiv.org/abs/2312.04000 · HTML: https://mlsanity.com/paper/2312.04000/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Understanding Collapse in Non-Contrastive Siamese Representation Learning](https://mlsanity.com/paper/2209.15007/) (2022)
- [What shapes the loss landscape of self-supervised learning?](https://mlsanity.com/paper/2210.00638/) (2022)

## Built on by (most cited first, 27 of 27)

- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Vision-Language Models Provide Promptable Representations for Reinforcement Learning](https://mlsanity.com/paper/2402.02651/) (2024)
- [Tracing the Representation Geometry of Language Models from Pretraining to Post-training](https://mlsanity.com/paper/2509.23024/) (2025)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)
- [How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks](https://mlsanity.com/paper/2407.03475/) (2024)
- [Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers](https://mlsanity.com/paper/2509.24317/) (2025)
- [stable-pretraining-v1: Foundation Model Research Made Simple](https://mlsanity.com/paper/2511.19484/) (2025)
- [Harnessing small projectors and multiple views for efficient vision pretraining](https://mlsanity.com/paper/2312.10725/) (2023)
- [Why and How Auxiliary Tasks Improve JEPA Representations](https://mlsanity.com/paper/2509.12249/) (2025)
- [Exploring Structural Degradation in Dense Representations for Self-supervised Learning](https://mlsanity.com/paper/2510.17299/) (2025)
- [AdaDim: Dimensionality Adaptation for SSL Representational Dynamics](https://mlsanity.com/paper/2505.12576/) (2025)
- [Geometric Metrics and LLMs: What They Measure and When They Work](https://mlsanity.com/paper/2509.25359/) (2025)
- [Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications](https://mlsanity.com/paper/2403.18316/) (2024)
- [Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions](https://mlsanity.com/paper/2505.14543/) (2025)
- [Quantifying Compositionality of Classic and State-of-the-Art Embeddings](https://mlsanity.com/paper/2509.19332/) (2025)
- [IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension](https://mlsanity.com/paper/2606.03338/) (2026)
- [Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning](https://mlsanity.com/paper/2410.10773/) (2024)
- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.15263/) (2026)
- [Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models](https://mlsanity.com/paper/2604.10949/) (2026)
- [Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles](https://mlsanity.com/paper/2606.09718/) (2026)
- [Task Priors: Enhancing Model Evaluation by Considering the Entire Space of Downstream Tasks](https://mlsanity.com/paper/2507.09871/) (2025)
- [Elucidating the Role of Feature Normalization in IJEPA](https://mlsanity.com/paper/2508.02829/) (2025)
- [Radial Basis Function Networks as Projection Heads in Self-Supervised Learning](https://mlsanity.com/paper/2606.21590/) (2026)
- [What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models](https://mlsanity.com/paper/2608.14819/) (2026)
- [A JEPA Recipe for Tabular Foundation Models](https://mlsanity.com/paper/2609.25541/) (2026)

## Abstract (arXiv)

Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent challenge of evaluating learned representations without access to a downstream task, and an annotated dataset. Without efficient and reliable evaluation, it is difficult to iterate on architectural and training choices for JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the quality of representations within JE architectures. Our metric addresses several shortcomings of recent approaches based on feature covariance rank by discriminating between informative and uninformative features. In essence, LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix associated with the surrogate SSL task -- a measure that intuitively captures the information content as it pertains to solving the SSL task. We empirically demonstrate that LiDAR significantly surpasses naive rank based approaches in its predictive power of optimal hyperparameters. Our proposed criterion presents a more robust and intuitive means of assessing the quality of representations within JE architectures, which we hope facilitates broader adoption of these powerful techniques in various domains.