mlsanityData to 7 Oct 2026

ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning

arXiv 2502.16268 ↗First posted 22 Feb 2025cs.CLcited by 17

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Evaluating large language models (LLMs) poses significant challenges, particularly due to issues of data contamination and the leakage of correct answers. To address these challenges, we introduce ThinkBench, a novel evaluation framework designed to evaluate LLMs' reasoning capability robustly. ThinkBench proposes a dynamic data generation method for constructing out-of-distribution (OOD) datasets and offers an OOD dataset that contains 2,912 samples drawn from reasoning tasks. ThinkBench unifies the evaluation of reasoning models and non-reasoning models. We evaluate 16 LLMs and 4 PRMs under identical experimental conditions and show that most of the LLMs' performance are far from robust and they face a certain level of data leakage. By dynamically generating OOD datasets, ThinkBench effectively provides a reliable evaluation of LLMs and reduces the impact of data contamination.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 71 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

38 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeek-V3 Technical Report27 Dec
  2. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models12 Oct
  3. 2022Making Large Language Models Better Reasoners with Step-Aware Verifier6 Jun

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Mistral 7Bcited by 3,065
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2024Mixtral of Expertscited by 1,785
23 more
  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  8. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  10. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  11. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  12. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  13. 2022Holistic Evaluation of Language Modelscited by 115
  14. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  15. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  16. 2024Not All LLM Reasoners Are Created Equalcited by 29
  17. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 26
  18. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  19. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Modelscited by 19
  20. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizabilitycited by 18
  21. 2021Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutionscited by 16
  22. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  23. 2024OpenAI-o1 AB Testing: Does the o1 model really do good reasoning in math problem solving?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 4.

17 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  4. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 7
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  6. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  7. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 6
  8. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  9. 2025MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMscited by 3
  10. 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMscited by 3
  11. 2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
  12. 2025Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentationcited by 2
5 more
  1. 2025Foundation models may exhibit staged progression in novel CBRN threat disclosurecited by 1
  2. 2026Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solvingcited by 0
  3. 2026Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarkscited by 0
  4. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0
  5. 2026Semifactual Credit-Augmented Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Are Your LLMs Capable of Stable Reasoning?
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
  3. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  5. 2026HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs
  6. 2025CER: Confidence Enhanced Reasoning in LLMs
  7. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
  8. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models