mlsanityData to 7 Oct 2026

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

arXiv 2410.14641 ↗First posted 18 Oct 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

24 of 36 references.

References · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  12. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
12 more
  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  4. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  5. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  6. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  7. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  8. 2022When Language Model Meets Private Librarycited by 54
  9. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  10. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  11. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  12. 2023Empirical Evaluation of ChatGPT on Requirements Information Retrieval Under Zero-Shot Settingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  2. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  3. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  4. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  5. 2025DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answeringcited by 3
  6. 2025Self-Review Framework for Enhancing Instruction Following Capability of LLMcited by 1
  7. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  8. 2026Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarkscited by 1
  9. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  10. 2026Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agentscited by 1
  11. 2026Agentic Interpretation: Lattice-Structured Evidence for LLM-Based Program Analysiscited by 0
  12. 2026Storyline Trees: Hierarchical Representations for Long-Form Narrativescited by 0
3 more
  1. 2026The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agentscited by 0
  2. 2026Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systemscited by 0
  3. 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agentscited by 0