mlsanityData to 14 Oct 2026

NoLiMa: Long-Context Evaluation Beyond Literal Matching

arXiv 2502.05167 ↗First posted 7 Feb 2025cs.CLcited by 126

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

30 of 44 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
18 more
  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2014Neural Turing Machinescited by 644
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  6. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  7. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  8. 2024Many-Shot In-Context Learningcited by 206
  9. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  10. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  11. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  12. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  13. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  14. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  15. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  16. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  17. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  18. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 90.

126 papers.

Sort:

By area:

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  2. 2026MiMo-V2-Flash Technical Reportcited by 191
  3. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  4. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 60
  5. 2025The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Managementcited by 48
  6. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  7. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 26
  8. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 23
  9. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  10. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversationscited by 20
  11. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 18
  12. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
38 more
  1. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  2. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 13
  3. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  4. 2025TextQuests: How Good are LLMs at Text-Based Video Games?cited by 12
  5. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  6. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  7. 2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?cited by 11
  8. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  9. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 9
  10. 2026AgentSwing: Adaptive Parallel Context Management Routing for Long-Horizon Web Agentscited by 9
  11. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  12. 2025ContestTrade: A Multi-Agent Trading System Based on Internal Contest Mechanismcited by 8
  13. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  14. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  15. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  16. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 7
  17. 2026Harnessing AtomisticSkills for Agentic Atomistic Researchcited by 6
  18. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 6
  19. 2026MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modelingcited by 5
  20. 2026Parallel Context Compaction for Long-Horizon LLM Agent Servingcited by 5
  21. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  22. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  23. 2025Reasoning Models Reason Well, Until They Don'tcited by 4
  24. 2026EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memorycited by 4
  25. 2026Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancingcited by 4
  26. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  27. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoningcited by 4
  28. 2026Context as an Environment: Programmatic Context Management for Long-Horizon Agentscited by 4
  29. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Modelscited by 3
  30. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  31. 2025CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Densitycited by 3
  32. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  33. 2025KForge: Program Synthesis for Diverse AI Hardware Acceleratorscited by 3
  34. 2025Training-Free Active Learning Framework in Materials Science with Large Language Modelscited by 3
  35. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  36. 2026Do Reasoning LLMs Refuse What They Infer in Long Contexts?cited by 3
  37. 2026Diagnosing and Mitigating Context Rot in Long-horizon Searchcited by 3
  38. 2025Insight-RAG: Enhancing LLMs with Insight-Driven Augmentationcited by 2

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
  2. 2025Reasoning on Multiple Needles In A Haystack
  3. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
  4. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoning
  5. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Models
  6. 2024Needle In A Multimodal Haystack
  7. 2026Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale
  8. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data