Benchmark-Driven Selection of AI: Evidence from DeepSeek-R1
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Evaluation of reasoning language models gained importance after it was observed that they can combine their existing capabilities into novel traces of intermediate steps before task completion and that the traces can sometimes help them to generalize better than past models. As reasoning becomes the next scaling dimension of large language models, careful study of their capabilities in critical tasks is needed. We show that better performance is not always caused by test-time algorithmic improvements or model sizes but also by using impactful benchmarks as curricula for learning. We call this benchmark-driven selection of AI and show its effects on DeepSeek-R1 using our sequential decision-making problem from Humanity's Last Exam. Steering development of AI by impactful benchmarks trades evaluation for learning and makes novelty of test tasks key for measuring generalization capabilities of reasoning models. Consequently, some benchmarks could be seen as curricula for training rather than unseen test sets.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 32 references that are in the map. First posted on arXiv 13 Aug 2025.
Builds on
20 of 32 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
- 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
- 2025Humanity's Last Examcited by 491
- 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
- 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
- 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
7 more
- 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
- 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
- 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 69
- 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
- 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
- 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
- 2025Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapointscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
- 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
- 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
- 2025General Scales Unlock AI Evaluation with Explanatory and Predictive Power
- 2025Benchmarking Reasoning Robustness in Large Language Models
- 2025Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens
- 2026Efficient Benchmarking of AI Agents
- 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models