mlsanityData to 14 Oct 2026

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

arXiv 2307.11088 ↗First posted 20 Jul 2023cs.CLcited by 233

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k~200k tokens). On the other hand, we investigate the effectiveness in evalution metrics for LCLMs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment, and thus we strongly advocate for length-instruction-enhanced (LIE) evaluation and employing LLM judges. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of more principled evaluation of these models.

Key facts. Cited by 233 papers in the map (data through 14 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

51 of 75 references.

References · 51

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
38 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  13. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  14. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  15. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  16. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  17. 2021Efficient Attentions for Long Document Summarizationcited by 345
  18. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  19. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  20. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  21. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  22. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  23. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  24. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  25. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  26. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  27. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  28. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Frameworkcited by 98
  29. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  30. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  31. 2020Extractive Opinion Summarization in Quantized Transformer Spacescited by 76
  32. 2021Can We Automate Scientific Reviewing?cited by 63
  33. 2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53
  34. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  35. 2023In-Context Learning with Many Demonstration Examplescited by 39
  36. 2017Supervised and Unsupervised Transfer Learning for Question Answeringcited by 24
  37. 2016Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machinecited by 23
  38. 2023Linearized Relative Positional Encodingcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 81; 2025: 70; 2026: 66.

233 papers.

Sort:

By area:

  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  8. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
38 more
  1. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  2. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  5. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  6. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  8. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  9. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  10. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  11. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  12. 2024Make Your LLM Fully Utilize the Contextcited by 119
  13. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  14. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  15. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  16. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 106
  17. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  18. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  19. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 84
  20. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  21. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  22. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  23. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  24. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 73
  25. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  26. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  27. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 64
  28. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  29. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  30. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  31. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  32. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  33. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  34. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 41
  35. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  36. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 36
  37. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 33
  38. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
  2. 2024Enhancing LLM Evaluations: The Garbling Trick
  3. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  4. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  5. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  6. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  7. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models
  8. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?