mlsanityData to 14 Oct 2026

LongEmbed: Extending Embedding Models for Long Context Retrieval

arXiv 2404.12096 ↗First posted 18 Apr 2024cs.CL cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

Embedding models play a pivot role in modern NLP applications such as IR and RAG. While the context limit of LLMs has been pushed beyond 1 million tokens, embedding models are still confined to a narrow context window not exceeding 8k tokens, refrained from application scenarios requiring long inputs such as legal contracts. This paper explores context window extension of existing embedding models, pushing the limit to 32k without requiring additional training. First, we examine the performance of current embedding models for long context retrieval on our newly constructed LongEmbed benchmark. LongEmbed comprises two synthetic tasks and four carefully chosen real-world tasks, featuring documents of varying length and dispersed target information. Benchmarking results underscore huge room for improvement in these models. Based on this, comprehensive experiments show that training-free context window extension strategies like position interpolation can effectively extend the context window of existing embedding models by several folds, regardless of their original context being 512 or beyond 4k. Furthermore, for models employing absolute position encoding (APE), we show the possibility of further fine-tuning to harvest notable performance gains while strictly preserving original behavior for short inputs. For models using rotary position embedding (RoPE), significant enhancements are observed when employing RoPE-specific methods, such as NTK and SelfExtend, indicating RoPE's superiority over APE for context window extension. To facilitate future research, we release E5-Base-4k and E5-RoPE-Base, along with the LongEmbed benchmark.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

49 of 59 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
37 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  11. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  12. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  13. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  14. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  15. 2023Improving Text Embeddings with Large Language Modelscited by 303
  16. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  17. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  18. 2024Generative Representational Instruction Tuningcited by 233
  19. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  20. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  21. 2023Augmenting Language Models with Long-Term Memorycited by 189
  22. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  23. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  24. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  25. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  26. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  27. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  28. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 104
  29. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  30. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  31. 2022Parallel Context Windows for Large Language Modelscited by 81
  32. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  33. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  34. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 27
  35. 2024Extending LLMs' Context Window with 100 Samplescited by 21
  36. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  37. 2024Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrievalcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 22; 2026: 15.

48 papers.

Sort:

By area:

  1. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  2. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  3. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  4. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 76
  5. 2024Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Modelscited by 73
  6. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  7. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  8. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  9. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  10. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 22
  11. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  12. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
36 more
  1. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  2. 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 18
  3. 2024Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understandingcited by 16
  4. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  5. 2024Length-Induced Embedding Collapse in PLM-based Modelscited by 9
  6. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  7. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  8. 2025Granite Embedding R2 Modelscited by 8
  9. 2025MIRB: Mathematical Information Retrieval Benchmarkcited by 6
  10. 2025Snoopy: Effective and Efficient Semantic Join Discovery via Proxy Columnscited by 4
  11. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  12. 2025Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledgecited by 3
  13. 2025Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddingscited by 3
  14. 2025Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPscited by 2
  15. 2026Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Biascited by 2
  16. 2024Pralekha: Cross-Lingual Document Alignment for Indic Languagescited by 1
  17. 2025Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical textscited by 1
  18. 2025ParaVul: A Parallel Large Language Model and Retrieval-Augmented Framework for Smart Contract Vulnerability Detectioncited by 1
  19. 2025FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fictioncited by 1
  20. 2026LMK > CLS: Landmark Pooling for Dense Embeddingscited by 1
  21. 2026Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrievalcited by 1
  22. 2024Lab-AI: Using Retrieval Augmentation to Enhance Language Models for Personalized Lab Test Interpretation in Clinical Medicinecited by 0
  23. 2025GeAR: Generation Augmented Retrievalcited by 0
  24. 2025Embedding-Based Context-Aware Rerankercited by 0
  25. 2025LLM2IR: simple unsupervised contrastive learning makes long-context LLM great retrievercited by 0
  26. 2026QueryPlot: Generating Geological Evidence Layers using Natural Language Queries for Mineral Explorationcited by 0
  27. 2026FGR-ColBERT: Identifying Fine-Grained Relevance Tokens During Retrievalcited by 0
  28. 2026Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Modelscited by 0
  29. 2026MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarkscited by 0
  30. 2026MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarioscited by 0
  31. 2026Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregationcited by 0
  32. 2026HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditionscited by 0
  33. 2026When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodingscited by 0
  34. 2026Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddingscited by 0
  35. 2026From Rules to Neural Graphs: Scalable Structured Prediction for Patent Prior Art Searchcited by 0
  36. 2026UltraBERT: Beyond the Limit of BERT Pre-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
  2. 2023Retrieval meets Long Context Large Language Models
  3. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  4. 2025Dewey Long Context Embedding Model: A Technical Report
  5. 2024ACER: Automatic Language Model Context Extension via Retrieval
  6. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  7. 2025SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension
  8. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models