mlsanityData to 14 Oct 2026

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks

arXiv 2412.15204 ↗First posted 19 Dec 2024cs.CL cs.AIcited by 379

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding. To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds. We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint. Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%. These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2. The project is available at https://longbench2.github.io.

Key facts. Cited by 379 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 51 of its 68 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

51 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries19 Sep
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack14 Jun

Other references · 49

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  10. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  12. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
37 more
  1. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  4. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  5. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  6. 2021Efficient Attentions for Long Document Summarizationcited by 345
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  9. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  10. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  11. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  13. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  14. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  15. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  16. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  17. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  18. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 152
  19. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  20. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  21. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  22. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  23. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  24. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  25. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  26. 2020MAVEN: A Massive General Domain Event Detection Datasetcited by 104
  27. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  28. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 81
  29. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 78
  30. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 77
  31. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  32. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  33. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 50
  34. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  35. 2024Teaching Large Language Models an Unseen Language on the Flycited by 30
  36. 2022Answering Complex Logical Queries on Knowledge Graphs via Query Computation Tree Optimizationcited by 29
  37. 2023VisKoP: Visual Knowledge oriented Programming for Interactive Knowledge Base Question Answeringcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 108; 2026: 270.

379 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 811
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  3. 2025Kimi K2: Open Agentic Intelligencecited by 567
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  5. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  7. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  8. 2026Qwen3.5-Omni Technical Reportcited by 174
  9. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  10. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  11. 2025Recursive Language Modelscited by 104
  12. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
38 more
  1. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49
  2. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  3. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoningcited by 39
  4. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  5. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  6. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 30
  7. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  8. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  9. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 29
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  11. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  12. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 23
  13. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  14. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 22
  15. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  16. 2025A Survey on LLM Mid-Trainingcited by 21
  17. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  18. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  19. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  20. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  21. 2026LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmarkcited by 19
  22. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 18
  23. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 18
  24. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  25. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 17
  26. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  27. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  28. 2025MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answeringcited by 16
  29. 2025Graph World Modelcited by 16
  30. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 15
  31. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  32. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 14
  33. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  34. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  35. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 14
  36. 2026LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growthcited by 14
  37. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
  38. 2026MemoBrain: Executive Memory as an Agentic Brain for Reasoningcited by 13

50 most cited of 379 All 379 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  2. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  3. 2026Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering
  4. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  5. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  6. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  7. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
  8. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding