mlsanityData to 7 Oct 2026

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance

arXiv 2605.22202 ↗First posted 21 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

40 of 59 references.

References · 40

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  6. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  7. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
28 more
  1. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  2. 2023Language Models Represent Space and Timecited by 306
  3. 2023Improving Text Embeddings with Large Language Modelscited by 295
  4. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  5. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  6. 2020Embedding-based Retrieval in Facebook Searchcited by 155
  7. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  8. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  9. 2022Analyzing Transformers in Embedding Spacecited by 131
  10. 2022The Geometry of Multilingual Language Model Representationscited by 94
  11. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  12. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Modelcited by 36
  13. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  14. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  15. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  16. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  17. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19
  18. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  19. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 9
  20. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 7
  21. 2022Exploring Dimensionality Reduction Techniques in Multilingual Transformerscited by 6
  22. 2023All Roads Lead to Rome? Exploring the Invariance of Transformers' Representationscited by 6
  23. 2024Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddingscited by 6
  24. 2025The Knowledge Microscope: Features as Better Analytical Lenses than Neuronscited by 5
  25. 2025Pruning Large Language Models by Identifying and Preserving Functional Networkscited by 4
  26. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  27. 2019Situating Sentence Embedders with Nearest Neighbor Overlapcited by 3
  28. 2024Exploring Intra and Inter-language Consistency in Embeddings with ICAcited by 2

Built on by

Not cited yet.