mlsanityData to 7 Oct 2026

Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

arXiv 2601.02023 ↗First posted 5 Jan 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Hallucination detection and mitigation

Abstract (arXiv)

As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora. We investigate how fact placement, corpus-level distributions, and anti-hallucination ("Don't Make It Up") prompts influence model behavior by introducing a model-agnostic extended needle-in-a-haystack benchmark designed for scalability, which we apply to evaluate Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and hallucination risk. We identify two critical failure modes: Distributional Collapse, where performance degrades significantly when evidence is dispersed; and a Safety Tax, where anti-hallucination prompts cause over-conservative refusal of present facts and evidence, sharply reducing accuracy. Our results suggest that many failures stem from ineffective context utilization, as models struggle to prioritize relevant information even when it is present. These findings highlight the need for model-specific robustness and effective context management to ensure reliable deployment in long-horizon agentic workflows.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

27 of 33 references.

References · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024DeepSeek-V3 Technical Reportcited by 724
  5. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  7. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  8. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  10. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  11. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  12. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
15 more
  1. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 43
  2. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  3. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  4. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 23
  5. 2024Long Context RAG Performance of Large Language Modelscited by 23
  6. 2024Lessons from Studying Two-Hop Latent Reasoningcited by 15
  7. 2024When Context Leads but Parametric Memory Follows in Large Language Modelscited by 10
  8. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  9. 2024LongIns: A Challenging Long-context Instruction-based Exam for LLMscited by 6
  10. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  11. 2025FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factualitycited by 5
  12. 2024DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?cited by 4
  13. 2024Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correctioncited by 3
  14. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Stepscited by 3
  15. 2024Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknowncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models
  2. 2024Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models
  3. 2025Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning
  4. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
  5. 2026From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
  6. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
  7. 2026Anchored Confabulation: Partial Evidence Non-Monotonically Amplifies Confident Hallucination in LLMs
  8. 2026Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations