mlsanityData to 7 Oct 2026

LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

arXiv 2402.05136 ↗First posted 6 Feb 2024cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

State-of-the-art large language models (LLMs) are now claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, resulting in biased evaluation. This paper introduces LV-Eval, a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. LV-Eval features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion, keyword and phrase replacement, and keyword-recall-based metric design. The advantages of LV-Eval include controllable evaluation across different context lengths, challenging test instances with confusing facts, mitigated knowledge leakage, and more objective evaluations. We evaluate 15 LLMs on LV-Eval and conduct ablation studies on the benchmarking techniques. The results reveal that: (i) Moonshot-v1 and recent large-scale open-source models, such as Qwen-2.5-72B and Llama-3.1-70B, achieve the highest performance on LV-Eval, particularly at lengths below 64k. (ii) Models exhibit distinct score trends. For example, GLM-4-9B-128k, Yi-6B-200k, and Llama3-8B-1M exhibit a relatively gentle degradation of performance, but their absolute performances may not necessarily be higher than those of LLMs with shorter context lengths. (iii) LLMs' performances can significantly degrade in the presence of confusing information, especially in the pressure test of "needle in a haystack". (iv) Issues related to knowledge leakage and inaccurate metrics introduce bias in evaluation, and these concerns are alleviated in LV-Eval. All datasets and evaluation codes are released at: https://github.com/infinigence/LVEval.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

27 of 40 references.

References · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
15 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  5. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2022Memorizing Transformerscited by 219
  8. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  9. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  10. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  11. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Frameworkcited by 98
  12. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  13. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  14. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  15. 2020DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applicationscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 28; 2026: 25.

80 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  5. 2025Qwen2.5-1M Technical Reportcited by 158
  6. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  7. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  9. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  10. 2024Evaluating Quantized Large Language Modelscited by 75
  11. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  12. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
38 more
  1. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  2. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  3. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  4. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  5. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  6. 2024Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Modelscited by 17
  7. 2024Distance between Relevant Information Pieces Causes Bias in Long-Context LLMscited by 15
  8. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  9. 2024MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokenscited by 13
  10. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
  11. 2024Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Modelscited by 13
  12. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  13. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  14. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  15. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  16. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  17. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  18. 2025Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novelscited by 9
  19. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  20. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  21. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoningcited by 8
  22. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contextscited by 8
  23. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  24. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  25. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7
  26. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  27. 2024Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stackscited by 6
  28. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  29. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  30. 2025Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Modelscited by 6
  31. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  32. 2025100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?cited by 5
  33. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 5
  34. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  35. 2024CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarioscited by 4
  36. 2024PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference Systemcited by 4
  37. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluationcited by 4
  38. 2024Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Taskscited by 3

50 most cited of 80 All 80 (JSON)