mlsanityData to 7 Oct 2026

Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

arXiv 2609.28290 ↗First posted 23 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Meaning identity (whether two sentences say the same thing after wording changes) is treated in retrieval and RAG as a geometric fact about independently encoded sentence vectors. We show that, for frozen off-the-shelf encoders and language models, it is not: identity is computed when both sentences share one forward pass, and is not a property of the embedding geometry those systems ship. On overlap-matched PAWS-X, purpose-built encoders (BGE, E5, GTE, MiniLM, E5-Mistral-7B) reach English confirm AUC only 0.55-0.65 (dense peak 0.70). Independently encoded last-token states of Llama 3, Mistral, and Qwen do no better; late fusion of the two vectors stays near chance. The same probe on a joint forward pass reaches 0.90-0.96 from 1.5B to 32B, collapses under partner shuffle, is mid-depth, saturates near 0.94 by 3B, and appears more weakly in GPT-2 XL (0.76). The gap holds beyond Llama-style models on other causal LMs, bidirectional encoders (DeBERTa, RoBERTa), and encoder-decoders (Flan-T5, T5, BART). Fixed or linear readers over frozen independent encodings never unlock identity; nonlinear pair readers recover part of it only on the full 49k-pair PAWS train split (0.68-0.87). Off-the-shelf rerankers split: BGE-reranker-large reaches 0.94, while MS-MARCO and Jina stay at 0.55-0.64. Independently trained families compute the same relation and a 1.5B joint reader can distill it from unlabelled teacher scores, while no linear function of the teachers own independent vectors can. Bi-encoders can be fine-tuned to fit PAWS (0.87-0.93), but transfer and STS-B suffer. Cosine compares wording neighbourhoods; identity is a cheap computed operator, not a property of either sentence vector.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 23 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

22 of 23 references.

References · 22

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  12. 2019Passage Re-ranking with BERTcited by 629
10 more
  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  4. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  5. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  6. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  7. 2024PARAPHRASUS : A Comprehensive Benchmark for Evaluating Paraphrase Detection Modelscited by 7
  8. 2026Differential syntactic and semantic encoding in LLMscited by 6
  9. 2025Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Modelscited by 5
  10. 2026On the Persistent Effects of Lexicality in Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories
  2. 2026On the Geometric Structure of Layer Updates in Deep Language Models
  3. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-training
  4. 2026The Lattice Representation Hypothesis of Large Language Models
  5. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representations
  6. 2026Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints
  7. 2026Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
  8. 2026RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding