# Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

arXiv: https://arxiv.org/abs/2601.02023 · HTML: https://mlsanity.com/paper/2601.02023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Placed in

- LLM reasoning / [Hallucination detection and mitigation](https://mlsanity.com/lr/hallucination-detection-and-mitigation/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K](https://mlsanity.com/paper/2402.05136/) (2024)
- [Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation](https://mlsanity.com/paper/2401.15449/) (2024)
- [NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities](https://mlsanity.com/paper/2407.11963/) (2024)
- [Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?](https://mlsanity.com/paper/2411.16679/) (2024)
- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [Long Context RAG Performance of Large Language Models](https://mlsanity.com/paper/2411.03538/) (2024)
- [Lessons from Studying Two-Hop Latent Reasoning](https://mlsanity.com/paper/2411.16353/) (2024)
- [When Context Leads but Parametric Memory Follows in Large Language Models](https://mlsanity.com/paper/2409.08435/) (2024)
- [Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts](https://mlsanity.com/paper/2504.04713/) (2025)
- [LongIns: A Challenging Long-context Instruction-based Exam for LLMs](https://mlsanity.com/paper/2406.17588/) (2024)
- [FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality](https://mlsanity.com/paper/2508.00109/) (2025)
- ["Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models](https://mlsanity.com/paper/2507.05424/) (2025)
- [DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?](https://mlsanity.com/paper/2406.12641/) (2024)
- [Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction](https://mlsanity.com/paper/2407.03651/) (2024)
- [Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps](https://mlsanity.com/paper/2410.04422/) (2024)
- [Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknown](https://mlsanity.com/paper/2411.15993/) (2024)

## Abstract (arXiv)

As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora. We investigate how fact placement, corpus-level distributions, and anti-hallucination ("Don't Make It Up") prompts influence model behavior by introducing a model-agnostic extended needle-in-a-haystack benchmark designed for scalability, which we apply to evaluate Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and hallucination risk. We identify two critical failure modes: Distributional Collapse, where performance degrades significantly when evidence is dispersed; and a Safety Tax, where anti-hallucination prompts cause over-conservative refusal of present facts and evidence, sharply reducing accuracy. Our results suggest that many failures stem from ineffective context utilization, as models struggle to prioritize relevant information even when it is present. These findings highlight the need for model-specific robustness and effective context management to ensure reliable deployment in long-horizon agentic workflows.