mlsanityData to 14 Oct 2026

Quantifying Positional Biases in Text Embedding Models

arXiv 2412.15241 ↗First posted 13 Dec 2024cs.CL cs.AI cs.IRcited by 4

Not on the map. Why?

Abstract (arXiv)

Embedding models are crucial for tasks in Information Retrieval (IR) and semantic similarity measurement, yet their handling of longer texts and associated positional biases remains underexplored. In this study, we investigate the impact of content position and input size on text embeddings. Our experiments reveal that embedding models, irrespective of their positional encoding mechanisms, disproportionately prioritize the beginning of an input. Ablation studies demonstrate that insertion of irrelevant text or removal at the start of a document reduces cosine similarity between altered and original embeddings by up to 12.3% more than ablations at the end. Regression analysis further confirms this bias, with sentence importance declining as position moves further from the start, even with with content-agnosticity. We hypothesize that this effect arises from pre-processing strategies and chosen positional encoding techniques. These findings quantify the sensitivity of retrieval systems and suggest a new lens towards embedding model robustness.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

21 of 27 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  10. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  11. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  12. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
9 more
  1. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 139
  2. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  3. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  4. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  5. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  6. 2021TWEAC: Transformer with Extendable QA Agent Classifierscited by 37
  7. 2017Semantic Structure and Interpretability of Word Embeddingscited by 21
  8. 2022Robust Textual Embedding against Word-level Adversarial Attackscited by 12
  9. 2022Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signalscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025SAGE: A Realistic Benchmark for Semantic Understandingcited by 6
  2. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  3. 2025ParaScopes: What do Language Models Activations Encode About Future Text?cited by 2
  4. 2026Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Biascited by 2