mlsanityData to 14 Oct 2026

Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design

arXiv 2506.04734 ↗First posted 5 Jun 2025cs.AI cs.CL cs.LGcited by 3

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. However, our study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably. Therefore, we advocate for the establishment of a more rigorous paradigm for model performance evaluation and present our empirical assessments of the Deepseek-R1-Distill series models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 25 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

14 of 25 references.

References · 14

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  7. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  8. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  9. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  10. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 74
  11. 2024Enhancing LLM Reasoning with Reward-guided Tree Searchcited by 58
  12. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Modelscited by 27
2 more
  1. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  2. 2025Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacementscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  2. 2025Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLMcited by 5
  3. 2026Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signalscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  2. 2025Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis
  3. 2025Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models
  4. 2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
  5. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarks
  6. 2024Are Your LLMs Capable of Stable Reasoning?
  7. 2024Steamroller Problems: An Evaluation of LLM Reasoning Capability with Automated Theorem Prover Strategies
  8. 2026Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning