mlsanityData to 13 Oct 2026

DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities

arXiv 2502.17807 ↗First posted 25 Feb 2025cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

We present DocPuzzle, a rigorously constructed benchmark for evaluating long-context reasoning capabilities in large language models (LLMs). This benchmark comprises 100 expert-level QA problems requiring multi-step reasoning over long real-world documents. To ensure the task quality and complexity, we implement a human-AI collaborative annotation-validation pipeline. DocPuzzle introduces an innovative evaluation framework that mitigates guessing bias through checklist-guided process analysis, establishing new standards for assessing reasoning capacities in LLMs. Our evaluation results show that: 1)Advanced slow-thinking reasoning models like o1-preview(69.7%) and DeepSeek-R1(66.3%) significantly outperform best general instruct models like Claude 3.5 Sonnet(57.7%); 2)Distilled reasoning models like DeepSeek-R1-Distill-Qwen-32B(41.3%) falls far behind the teacher model, suggesting challenges to maintain the generalization of reasoning capabilities relying solely on distillation.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 35 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

10 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DeepSeek-V3 Technical Report27 Dec

Other references · 9

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  2. 2024Qwen2.5 Technical Reportcited by 4,840
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  4. 2023Let's Verify Step by Stepcited by 3,672
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 738
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 663
  9. 2025Humanity's Last Examcited by 478

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  2. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  3. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
  2. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  3. 2026DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
  4. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision
  5. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  6. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  7. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  8. 2025LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation