# Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? arXiv: https://arxiv.org/abs/2502.19361 ยท HTML: https://mlsanity.com/paper/2502.19361/ ## Key facts - Cited by 44 papers in the map (data through 14 Oct 2026). - Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. - Builds on 20 of its 58 references that are in the map. - First posted on arXiv 26 Feb 2025. ## Placed in - LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/) ## Builds on (most cited first, 20 of 20) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024) - [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023) - [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023) - [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024) - [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024) - [AlphaMath Almost Zero: Process Supervision without Process](https://mlsanity.com/paper/2405.03553/) (2024) - [On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks](https://mlsanity.com/paper/2402.08115/) (2024) - [CriticBench: Benchmarking LLMs for Critique-Correct Reasoning](https://mlsanity.com/paper/2402.14809/) (2024) - [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023) - [MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series](https://mlsanity.com/paper/2405.19327/) (2024) - [GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models](https://mlsanity.com/paper/2406.14550/) (2024) - [Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models](https://mlsanity.com/paper/2411.07140/) (2024) - [Large Language Models Can Self-Correct with Key Condition Verification](https://mlsanity.com/paper/2405.14092/) (2024) - [Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic](https://mlsanity.com/paper/2408.16326/) (2024) - [Critique Ability of Large Language Models](https://mlsanity.com/paper/2310.04815/) (2023) - [AIR: Complex Instruction Generation via Automatic Iterative Refinement](https://mlsanity.com/paper/2502.17787/) (2025) ## Built on by (most cited first, 30 of 44) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025) - [OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs](https://mlsanity.com/paper/2510.10689/) (2025) - [xVerify: Efficient Answer Verifier for Reasoning Model Evaluations](https://mlsanity.com/paper/2504.10481/) (2025) - [Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective](https://mlsanity.com/paper/2505.12886/) (2025) - [Synergizing RAG and Reasoning: A Systematic Review](https://mlsanity.com/paper/2504.15909/) (2025) - [Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models](https://mlsanity.com/paper/2505.13774/) (2025) - [What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.22148/) (2025) - [CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization](https://mlsanity.com/paper/2507.06181/) (2025) - [CoRT: Code-integrated Reasoning within Thinking](https://mlsanity.com/paper/2506.09820/) (2025) - [Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems](https://mlsanity.com/paper/2505.17968/) (2025) - [WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality](https://mlsanity.com/paper/2510.18560/) (2025) - [SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models](https://mlsanity.com/paper/2511.05459/) (2025) - [Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Reading](https://mlsanity.com/paper/2504.09402/) (2025) - [Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps](https://mlsanity.com/paper/2503.19326/) (2025) - [DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMs](https://mlsanity.com/paper/2503.15793/) (2025) - [Graph of Verification: Structured Verification of LLM Reasoning with Directed Acyclic Graphs](https://mlsanity.com/paper/2506.12509/) (2025) - [RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback](https://mlsanity.com/paper/2507.15024/) (2025) - [TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning](https://mlsanity.com/paper/2512.13106/) (2025) - [Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization](https://mlsanity.com/paper/2507.02145/) (2025) - [R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge](https://mlsanity.com/paper/2508.00324/) (2025) - [Meta-R1: Empowering Large Reasoning Models with Metacognition](https://mlsanity.com/paper/2508.17291/) (2025) - [Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories](https://mlsanity.com/paper/2606.02060/) (2026) - [Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?](https://mlsanity.com/paper/2510.06410/) (2025) - [Teaching Language Models to Reason with Tools](https://mlsanity.com/paper/2510.20342/) (2025) - [Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization](https://mlsanity.com/paper/2601.04992/) (2026) - [LATTS: Locally Adaptive Test-Time Scaling](https://mlsanity.com/paper/2509.20368/) (2025) - [Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Models](https://mlsanity.com/paper/2509.24711/) (2025) - [Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation](https://mlsanity.com/paper/2510.07414/) (2025) - [Self-Verifying Reflection Helps Transformers with CoT Reasoning](https://mlsanity.com/paper/2510.12157/) (2025) - [Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility](https://mlsanity.com/paper/2601.13398/) (2026) ## Abstract (arXiv) Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.