# Anchor Divergence for Semantic Geometry in Contrastive Learning

arXiv: https://arxiv.org/abs/2610.06919 · HTML: https://mlsanity.com/paper/2610.06919/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [SuS-X: Training-Free Name-Only Transfer of Vision-Language Models](https://mlsanity.com/paper/2211.16198/) (2022)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [Meta CLIP 2: A Worldwide Scaling Recipe](https://mlsanity.com/paper/2507.22062/) (2025)
- [Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion](https://mlsanity.com/paper/2502.04263/) (2025)
- [Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification](https://mlsanity.com/paper/2404.17753/) (2024)
- [CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation](https://mlsanity.com/paper/2502.19842/) (2025)
- [Parts of Speech-Grounded Subspaces in Vision-Language Models](https://mlsanity.com/paper/2305.14053/) (2023)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [Leveraging Foundation Models for Content-Based Image Retrieval in Radiology](https://mlsanity.com/paper/2403.06567/) (2024)

## Abstract (arXiv)

This paper concerns how semantic context determines geometry in learned vector representations. Similarity is typically measured using cosine similarity, which provides a single fixed geometry. Semantic similarity, however, is inherently context dependent: two images may be similar because they depict the same object, share a visual style, or are relevant to the same clinical finding. We show that contrastive representations naturally encompass a family of geometries that can be specialized to particular semantic structure. The key idea is to use an interplay between contrastive learning, exponential families, and information geometry to establish a correspondence between probability distributions over "anchors" and Bregman geometries on the representation space. We use this correspondence to define "Anchor Divergences", a method for specifying context-specific semantic geometries on fixed representations. Under this correspondence, modeling the anchor distribution models the geometry itself. Experiments on retrieval show that anchor divergences provide an effective and efficient way to specify context-specific semantic similarity.