mlsanityData to 14 Oct 2026

RULER: What's the Real Context Size of Your Long-Context Language Models?

arXiv 2404.06654 ↗First posted 9 Apr 2024cs.CLcited by 1,346

Not on the map. Why?

Cited by 273 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

Key facts. Cited by 1,346 papers in the map (data through 14 Oct 2026). Builds on 59 of its 91 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

59 of 91 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  12. 2024Yi: Open Foundation Models by 01.AIcited by 719
  13. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  14. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  15. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  17. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  18. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  19. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  20. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  21. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  22. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  23. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  24. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  25. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  26. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  27. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  28. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  29. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  30. 2023Augmenting Language Models with Long-Term Memorycited by 189
  31. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  32. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  33. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  34. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  35. 2022A Length-Extrapolatable Transformercited by 127
  36. 2023Retrieval meets Long Context Large Language Modelscited by 115
  37. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  38. 2024Long Context Compression with Activation Beaconcited by 104

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 146; 2025: 388; 2026: 812.

1,346 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  6. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
38 more
  1. 2024Titans: Learning to Memorize at Test Timecited by 328
  2. 2025Memory in the Age of AI Agentscited by 303
  3. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  4. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  5. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  8. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  9. 2026MiMo-V2-Flash Technical Reportcited by 191
  10. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  11. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  12. 2025Qwen2.5-1M Technical Reportcited by 164
  13. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  14. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  15. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  16. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  17. 2025Test-Time Training Done Rightcited by 130
  18. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  19. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  20. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  21. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  22. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  23. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  24. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  25. 2024Make Your LLM Fully Utilize the Contextcited by 119
  26. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  27. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  28. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  29. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  30. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  31. 2025Recursive Language Modelscited by 104
  32. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  33. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  34. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  35. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  36. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  37. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  38. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90

50 most cited of 1,346 All 1,346 (JSON)