mlsanityData to 7 Oct 2026

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

arXiv 2406.14673 ↗First posted 20 Jun 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs' long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a "know but don't tell" phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
9 more
  1. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  2. 2014A Tutorial on Principal Component Analysiscited by 139
  3. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  4. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  5. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  6. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  7. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  8. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  9. 2023Primacy Effect of ChatGPTcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 7.

36 papers.

Sort:

By area:

  1. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  2. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?cited by 36
  3. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  4. 2025Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generationcited by 12
  5. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  6. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoningcited by 9
  7. 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Readingcited by 8
  8. 2025Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learningcited by 7
  9. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  10. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Modelscited by 5
  11. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  12. 2025Intent-Aware Schema Generation And Refinement For Literature Review Tablescited by 5
24 more
  1. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  2. 2025Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signalscited by 4
  3. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Stepscited by 3
  4. 2025A Comparative Analysis of Contextual Representation Flow in State-Space and Transformer Architecturescited by 3
  5. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysiscited by 3
  6. 2024FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrievalcited by 2
  7. 2025EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generationcited by 2
  8. 2025Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understandingcited by 2
  9. 2025Benchmarking LLM-based agents for single-cell omics analysiscited by 2
  10. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  11. 2024Pay Attention to What Matterscited by 1
  12. 2024Membership Inference Attack against Long-Context Large Language Modelscited by 1
  13. 2025Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generationcited by 1
  14. 2025MemoCoder: Automated Function Synthesis using LLM-Supported Agentscited by 1
  15. 2025Mitigating Coordinate Prediction Bias from Positional Encoding Failurescited by 1
  16. 2026How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understandingcited by 1
  17. 2026TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Taskscited by 1
  18. 2025Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decodingcited by 0
  19. 2025Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Modelscited by 0
  20. 2025Evaluating the Challenges of LLMs in Real-world Medical Follow-up: A Comparative Study and An Optimized Frameworkcited by 0
  21. 2025The Instruction Gap: LLMs get lost in Following Instructioncited by 0
  22. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  23. 2026Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMscited by 0
  24. 2026SchemaLink: An Intelligent Web Editor for LinkML Schema Curationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  2. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
  3. 2026LLM Reasoning Is Latent, Not the Chain of Thought
  4. 2025A Survey on Transformer Context Extension: Approaches and Evaluation
  5. 2026Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
  6. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Models
  7. 2026A Decomposition Perspective to Long-context Reasoning for LLMs
  8. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs