# Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

arXiv: https://arxiv.org/abs/2406.14673 · HTML: https://mlsanity.com/paper/2406.14673/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding](https://mlsanity.com/paper/2403.04797/) (2024)
- [Do Androids Know They're Only Dreaming of Electric Sheep?](https://mlsanity.com/paper/2312.17249/) (2023)
- [How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study](https://mlsanity.com/paper/2402.16061/) (2024)
- [Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?](https://mlsanity.com/paper/2312.03729/) (2023)
- [Primacy Effect of ChatGPT](https://mlsanity.com/paper/2310.13206/) (2023)

## Built on by (most cited first, 30 of 36)

- [MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation](https://mlsanity.com/paper/2410.11779/) (2024)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [AD-LLM: Benchmarking Large Language Models for Anomaly Detection](https://mlsanity.com/paper/2412.11142/) (2024)
- [Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.19754/) (2025)
- [A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?](https://mlsanity.com/paper/2411.01414/) (2024)
- [Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Reading](https://mlsanity.com/paper/2504.09402/) (2025)
- [Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning](https://mlsanity.com/paper/2502.05675/) (2025)
- [Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning](https://mlsanity.com/paper/2502.12477/) (2025)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2505.19498/) (2025)
- ["Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models](https://mlsanity.com/paper/2507.05424/) (2025)
- [Intent-Aware Schema Generation And Refinement For Literature Review Tables](https://mlsanity.com/paper/2507.19521/) (2025)
- [ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions](https://mlsanity.com/paper/2605.24279/) (2026)
- [Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals](https://mlsanity.com/paper/2502.01025/) (2025)
- [Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps](https://mlsanity.com/paper/2410.04422/) (2024)
- [Contextual Representation Flow Across Depth in State-Space and Transformer Architectures](https://mlsanity.com/paper/2510.06640/) (2025)
- [ConDABench: Interactive Evaluation of Language Models for Data Analysis](https://mlsanity.com/paper/2510.13835/) (2025)
- [FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval](https://mlsanity.com/paper/2410.21012/) (2024)
- [EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generation](https://mlsanity.com/paper/2505.12424/) (2025)
- [Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding](https://mlsanity.com/paper/2505.13353/) (2025)
- [Benchmarking LLM-based agents for single-cell omics analysis](https://mlsanity.com/paper/2508.13201/) (2025)
- [Mitigating Coordinate Prediction Bias from Positional Encoding Failures](https://mlsanity.com/paper/2510.22102/) (2025)
- [DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models](https://mlsanity.com/paper/2602.22175/) (2026)
- [Pay Attention to What Matters](https://mlsanity.com/paper/2409.19001/) (2024)
- [Membership Inference Attack against Long-Context Large Language Models](https://mlsanity.com/paper/2411.11424/) (2024)
- [Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.05939/) (2025)
- [MemoCoder: Automated Function Synthesis using LLM-Supported Agents](https://mlsanity.com/paper/2507.18812/) (2025)
- [How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding](https://mlsanity.com/paper/2603.18009/) (2026)
- [TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks](https://mlsanity.com/paper/2604.06765/) (2026)
- [Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding](https://mlsanity.com/paper/2506.08371/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs' long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a "know but don't tell" phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.