# Harnessing the Universal Geometry of Embeddings

arXiv: https://arxiv.org/abs/2505.12540 · HTML: https://mlsanity.com/paper/2505.12540/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 71 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Grounding Language Models to Images for Multimodal Inputs and Outputs](https://mlsanity.com/paper/2301.13823/) (2023)
- [Gromov-Wasserstein Alignment of Word Embedding Spaces](https://mlsanity.com/paper/1809.00013/) (2018)

## Built on by (most cited first, 30 of 55)

- [Correlated Errors in Large Language Models](https://mlsanity.com/paper/2506.07962/) (2025)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [Transferring Linear Features Across Language Models With Model Stitching](https://mlsanity.com/paper/2506.06609/) (2025)
- [The Geometry of Reasoning: Flowing Logics in Representation Space](https://mlsanity.com/paper/2510.09782/) (2025)
- [Latent Space Communication via K-V Cache Alignment](https://mlsanity.com/paper/2601.06123/) (2026)
- [Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale](https://mlsanity.com/paper/2604.18572/) (2026)
- [Universally Converging Representations of Matter Across Scientific Foundation Models](https://mlsanity.com/paper/2512.03750/) (2025)
- [Jailbreak Transferability Emerges from Shared Representations](https://mlsanity.com/paper/2506.12913/) (2025)
- [DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection](https://mlsanity.com/paper/2507.15042/) (2025)
- [Dynamic Reflections: Probing Video Representations with Text Alignment](https://mlsanity.com/paper/2511.02767/) (2025)
- [Multi-Way Representation Alignment](https://mlsanity.com/paper/2602.06205/) (2026)
- [mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformations](https://mlsanity.com/paper/2510.02348/) (2025)
- [Learning Network Sheaves for AI-native Semantic Communication](https://mlsanity.com/paper/2512.03248/) (2025)
- [The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale](https://mlsanity.com/paper/2607.14144/) (2026)
- [Atlas-Alignment: Making Interpretability Transferable Across Language Models](https://mlsanity.com/paper/2510.27413/) (2025)
- [SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport](https://mlsanity.com/paper/2602.23353/) (2026)
- [The Universal Normal Embedding](https://mlsanity.com/paper/2603.21786/) (2026)
- [Shared representations in brains and models reveal a two-route cortical organization during scene perception](https://mlsanity.com/paper/2507.13941/) (2025)
- [ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Models](https://mlsanity.com/paper/2510.11278/) (2025)
- [Position: Require Frontier AI Labs To Release Small "Analog" Models](https://mlsanity.com/paper/2510.14053/) (2025)
- [Unique Lives, Shared World: Learning from Single-Life Videos](https://mlsanity.com/paper/2512.04085/) (2025)
- [Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks](https://mlsanity.com/paper/2601.18617/) (2026)
- [Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation](https://mlsanity.com/paper/2606.05290/) (2026)
- [Language Models Represent and Transform Concepts with Shared Geometry](https://mlsanity.com/paper/2607.04525/) (2026)
- [The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding](https://mlsanity.com/paper/2609.10296/) (2026)
- [Latent Reasoning via Sentence Embedding Prediction](https://mlsanity.com/paper/2505.22202/) (2025)
- [MOD-X: A Modular Open Decentralized eXchange Framework proposal for Heterogeneous Interoperable Artificial Intelligence Agents](https://mlsanity.com/paper/2507.04376/) (2025)
- [Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed](https://mlsanity.com/paper/2510.01494/) (2025)
- [Mapping Semantic & Syntactic Relationships with Geometric Rotation](https://mlsanity.com/paper/2510.09790/) (2025)
- [Exploring possible vector systems for faster training of neural networks with preconfigured latent spaces](https://mlsanity.com/paper/2512.07509/) (2025)

## Abstract (arXiv)

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.