mlsanityData to 7 Oct 2026

Evaluating Step-by-step Reasoning Traces: A Survey

arXiv 2502.12289 ↗First posted 17 Feb 2025cs.CLcited by 62

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 101 of its 136 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

101 of 136 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains11 Oct
  2. 2024MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs20 Jun
  3. 2024Deductive Beam Search: Decoding Deducible Rationale for Chain-of-Thought Reasoning31 Jan
  4. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation28 Dec
  5. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes22 Dec
  6. 2023Deductive Verification of Chain-of-Thought Reasoning6 Jun
  7. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness21 Apr
  8. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning15 Dec
  9. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  10. 2022Making Large Language Models Better Reasoners with Step-Aware Verifier6 Jun

Other references · 91

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  13. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  14. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  15. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  16. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  17. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  18. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  19. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  20. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  21. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  22. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  23. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  24. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  25. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  26. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  27. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  28. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  29. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  30. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  31. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  32. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  33. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  34. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  35. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  36. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  37. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  38. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 35.

62 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  3. 2025Probabilistic Soundness Guarantees in LLM Reasoning Chainscited by 13
  4. 2025ReasoningFlow: Semantic Structure of Complex Reasoning Tracescited by 12
  5. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 12
  6. 2025From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculationscited by 11
  7. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 11
  8. 2025What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluationcited by 10
  9. 2025The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Modelscited by 9
  10. 2025What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignmentcited by 7
  11. 2025Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoningcited by 7
  12. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarizationcited by 6
38 more
  1. 2025Humans Perceive Wrong Narratives from AI Reasoning Textscited by 6
  2. 2025Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Provingcited by 6
  3. 2025CLATTER: Comprehensive Entailment Reasoning for Hallucination Detectioncited by 5
  4. 2025Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoningcited by 5
  5. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5
  6. 2026Measuring Weak-to-Strong Legibility of Reasoning Modelscited by 5
  7. 2025CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentationcited by 4
  8. 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubricscited by 4
  9. 2026Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuningcited by 4
  10. 2025T-FIX: Text-Based Explanations with Features Interpretable to eXpertscited by 2
  11. 2025An Empirical Study of Reasoning Steps in Thinking Code LLMscited by 2
  12. 2026Reasoning Fails Where Step Flow Breakscited by 2
  13. 2025Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interactioncited by 1
  14. 2025DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoningcited by 1
  15. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 1
  16. 2026MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoningcited by 1
  17. 2026LLMs for Game Theory: Entropy-Guided In-Context Learning and Adaptive CoT Reasoningcited by 1
  18. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMscited by 1
  19. 2026Human-Inspired Continuous Learning of Internal Reasoning Processes: Learning How to Think for Adaptive AI Systemscited by 1
  20. 2026Directional Reasoning Trajectory Change (DRTC): Identifying Critical Trace Segments in Reasoning Modelscited by 1
  21. 2026A Benchmark for Deep Information Synthesiscited by 1
  22. 2026One-Token Verification for Reasoning Correctness Estimationcited by 1
  23. 2026Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?cited by 1
  24. 2026MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rayscited by 1
  25. 2026What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systemscited by 1
  26. 2026Can LLM Coding Agents Reason About Time Series?cited by 1
  27. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  28. 2026FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Constructioncited by 1
  29. 2026Evaluation of Multi-Turn Consistency in LLM Agents: Survival Analysis and Failure-Rationale Taxonomycited by 1
  30. 2025Dissecting Physics Reasoning in Small Language Models: A Multi-Dimensional Analysis from an Educational Perspectivecited by 0
  31. 2025PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMscited by 0
  32. 2025Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensicscited by 0
  33. 2025Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoringcited by 0
  34. 2025Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Errorcited by 0
  35. 2026KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learningcited by 0
  36. 2026Beyond Model Scaling: Test-Time Intervention for Efficient Deep Reasoningcited by 0
  37. 2026EvalQReason: A Framework for Step-Level Reasoning Evaluation in Large Language Modelscited by 0
  38. 2026When Scaling Fails: Mitigating Audio Perception Decay of LALMs via Multi-Step Perception-Aware Reasoningcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Assessing LLM Reasoning Steps via Principal Knowledge Grounding
  2. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  3. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure
  4. 2026LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator
  5. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  6. 2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique
  7. 2026TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
  8. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models