# Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB

arXiv: https://arxiv.org/abs/2511.11041 · HTML: https://mlsanity.com/paper/2511.11041/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Language-Driven Anchors for Zero-Shot Adversarial Robustness](https://mlsanity.com/paper/2301.13096/) (2023)
- [Decipher the Modality Gap in Multimodal Contrastive Learning: From Convergent Representations to Pairwise Alignment](https://mlsanity.com/paper/2510.03268/) (2025)
- [Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks](https://mlsanity.com/paper/2508.17744/) (2025)
- [Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings](https://mlsanity.com/paper/2305.10786/) (2023)
- [Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models](https://mlsanity.com/paper/2501.18280/) (2025)
- [Isotropy, Clusters, and Classifiers](https://mlsanity.com/paper/2402.03191/) (2024)
- [Contextual Subspace Manifold Projection for Structural Refinement of Large Language Model Representations](https://mlsanity.com/paper/2502.08026/) (2025)
- [Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply](https://mlsanity.com/paper/2509.01606/) (2025)

## Built on by (most cited first, 1 of 1)

- [Bringing Agentic Search to Earth Observation Data Discovery](https://mlsanity.com/paper/2607.02387/) (2026)

## Abstract (arXiv)

We find that current sentence-embedding models produce outputs with a consistent bias: every embedding e decomposes as tilde e + μ, where the mean μ is near-identical across all sentences. We study two training-free corrections -- subtracting μ directly (R1), or projecting each embedding off the mean direction (R2) -- and show, via a first-order error-propagation argument, that R2 cancels the parallel component of mean-estimation error that R1 retains. Across 38 models on the Massive Multilingual Text Embedding Benchmark (MMTEB)~citepMMTEB, R2 yields consistent classification gains (paired bar t = 3.31, 29 of 38 models with t>2, zero losses), and the per-model mean norm VertμVert correlates with which models benefit most. A nine-method dose-response ablation on five models further reveals that mild single-direction removal helps, but full principal component analysis (PCA) whitening hurts every model we test, and that R2 and All-but-the-Top with depth one agree within 0.18 pp downstream despite weak geometric alignment between hatμ and the centered top principal component.