# Geometric Analysis of Self-Supervised Vision Representations for Semantic Image Retrieval

arXiv: https://arxiv.org/abs/2604.24469 · HTML: https://mlsanity.com/paper/2604.24469/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [A Decade Survey of Content Based Image Retrieval using Deep Learning](https://mlsanity.com/paper/2012.00641/) (2020)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [InsCLR: Improving Instance Retrieval with Self-Supervision](https://mlsanity.com/paper/2112.01390/) (2021)
- [Clustering Properties of Self-Supervised Learning](https://mlsanity.com/paper/2501.18452/) (2025)
- [Leveraging Foundation Models for Content-Based Image Retrieval in Radiology](https://mlsanity.com/paper/2403.06567/) (2024)
- [Evaluating Contrastive Models for Instance-based Image Retrieval](https://mlsanity.com/paper/2104.14939/) (2021)
- [VG-SSL: Benchmarking Self-supervised Representation Learning Approaches for Visual Geo-localization](https://mlsanity.com/paper/2308.00090/) (2023)
- [Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning](https://mlsanity.com/paper/2603.03530/) (2026)
- [Advancements in Content-Based Image Retrieval: A Comprehensive Survey of Relevance Feedback Techniques](https://mlsanity.com/paper/2312.10089/) (2023)
- [Global Geometry Is Not Enough for Vision Representations](https://mlsanity.com/paper/2602.03282/) (2026)
- [Hypersolid: Emergent Vision Representations via Short-Range Repulsion](https://mlsanity.com/paper/2601.21255/) (2026)

## Abstract (arXiv)

Content-based image retrieval (CBIR) systems enable users to search images based on visual content instead of relying on metadata. The text domain has benefited from vector search of representations created with unsupervised methods such as BERT. However, modern self-supervised learning methods for vision are mostly not reported in CBIR-related literature, instead relying on supervised models or multi-modal methods that align text and vision. We evaluate how the representations learned by modern self-supervised learning methods for vision perform under typical retrieval stacks that leverage vector databases and nearest neighbor search. Our evaluation reveals that the latent space geometry impacts approximate nearest neighbor (ANN) indexing. Specifically, highly anisotropic representations with high skewness produced by several modern SSL methods degrade the performance of partition-based and hashing-based search, even if their own linear probe or K-NN accuracy is not affected. In contrast, representations with higher isotropy and local purity better satisfy the distance-based assumptions of ANN indexes, leading to improved semantic retrieval performance.