mlsanityData to 14 Oct 2026

HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

arXiv 2410.02694 ↗First posted 3 Oct 2024cs.CL cs.AIcited by 125

Not on the map. Why?

Abstract (arXiv)

Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the underlying reasons behind these practices and find that existing benchmarks often provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions -- the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Builds on 63 of its 75 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

63 of 75 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  9. 2024Yi: Open Foundation Models by 01.AIcited by 719
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  13. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  15. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  16. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  17. 2023Larger language models do in-context learning differentlycited by 394
  18. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  19. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  20. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  21. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  22. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  23. 2020Overview of the TREC 2019 deep learning trackcited by 286
  24. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  25. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  26. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  27. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  28. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  29. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  30. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  31. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  32. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  33. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  34. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  35. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  36. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  37. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 112
  38. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 58; 2026: 58.

125 papers.

Sort:

By area:

  1. 2024Phi-4 Technical Reportcited by 711
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  3. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  4. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  5. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  7. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  8. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 61
  9. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  10. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  11. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 32
  12. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
38 more
  1. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  2. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 29
  3. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 27
  4. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  5. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kcited by 21
  6. 2025A Survey on LLM Mid-Trainingcited by 21
  7. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  8. 2026LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmarkcited by 19
  9. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 18
  10. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 17
  11. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  12. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  13. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  14. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  15. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
  16. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  17. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  18. 2025Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Searchcited by 12
  19. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 11
  20. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  21. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 10
  22. 2025Self-Taught Agentic Long Context Understandingcited by 10
  23. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  24. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 10
  25. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  26. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 9
  27. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 9
  28. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 9
  29. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 9
  30. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  31. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  32. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problemscited by 7
  33. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  34. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  35. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  36. 2025PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contextscited by 7
  37. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 7
  38. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6

50 most cited of 125 All 125 (JSON)