mlsanityData to 7 Oct 2026

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

arXiv 2404.04163 ↗First posted 5 Apr 2024cs.IR cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

35 of 49 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
23 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  7. 2023Adapting Language Models to Compress Contextscited by 350
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  9. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  10. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  11. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  12. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  13. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  14. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  15. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  16. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
  17. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  18. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  19. 2024User Embedding Model for Personalized Language Promptingcited by 28
  20. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  21. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  22. 2023DAPR: A Benchmark on Document-Aware Passage Retrievalcited by 9
  23. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  2. 2024StateAct: Enhancing LLM Base Agents via Self-prompting and State-trackingcited by 21
  3. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  4. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  5. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  6. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generationcited by 9
  7. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 4
  8. 2026Answer Bubbles: Information Exposure in AI-Mediated Searchcited by 3
  9. 2026PosIR: Position-Aware Heterogeneous Information Retrieval Benchmarkcited by 2
  10. 2026Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Biascited by 2
  11. 2025ImpliRet: Benchmarking the Implicit Fact Retrieval Challengecited by 1
  12. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
1 more
  1. 2026Masking or Mitigating? Deconstructing the Impact of Query Rewriting on Retriever Biases in RAGcited by 0