mlsanityData to 7 Oct 2026

A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains

arXiv 2402.00559 ↗First posted 1 Feb 2024cs.CLcited by 63

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Prompting language models to provide step-by-step answers (e.g., "Chain-of-Thought") is the prominent approach for complex reasoning tasks, where more accurate reasoning chains typically improve downstream task performance. Recent literature discusses automatic methods to verify reasoning to evaluate and improve their correctness. However, no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods, hindering progress in this direction. We introduce REVEAL: Reasoning Verification Evaluation, a dataset to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings. REVEAL includes comprehensive labels for the relevance, attribution to evidence passages, and logical correctness of each reasoning step in a language model's answer, across a variety of datasets and state-of-the-art language models. Evaluation on REVEAL shows that verifiers struggle at verifying reasoning chains - in particular, verifying logical correctness and detecting contradictions. Available at https://reveal-dataset.github.io/ .

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 64 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

37 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ThoughtSource: A central hub for large language model reasoning data27 Jan
  2. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations24 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  12. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
22 more
  1. 2021A Survey on Automated Fact-Checkingcited by 376
  2. 2021True Few-Shot Learning with Language Modelscited by 343
  3. 2023How Language Model Hallucinations Can Snowballcited by 310
  4. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  5. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  6. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  7. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  8. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  9. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  10. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  11. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  12. 2021Explaining Answers with Entailment Treescited by 148
  13. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  14. 2021Decontextualization: Making Sentences Stand-Alonecited by 101
  15. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  16. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  17. 2018Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detectioncited by 55
  18. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  19. 2020Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMRcited by 25
  20. 2021What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?cited by 23
  21. 2021How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AIcited by 22
  22. 2023A Comprehensive Evaluation of Tool-Assisted Generation Strategiescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 29.

63 papers.

Sort:

By area:

  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  2. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  3. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  4. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  5. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  6. 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 39
  7. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  8. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Frameworkcited by 16
  9. 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Datacited by 14
  10. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
  11. 2024CoDA: Instructive Chain-of-Domain Adaptation with Severity-Aware Visual Prompt Tuningcited by 13
  12. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
38 more
  1. 2025GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checkingcited by 12
  2. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
  3. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  4. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  5. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 9
  6. 2024Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Stepcited by 8
  7. 2024Rationale-Aware Answer Verification by Pairwise Self-Evaluationcited by 8
  8. 2024Defining Knowledge: Bridging Epistemology and Large Language Modelscited by 7
  9. 2024FLARE: Faithful Logic-Aided Reasoning and Explorationcited by 7
  10. 2026Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoningcited by 7
  11. 2024CoverBench: A Challenging Benchmark for Complex Claim Verificationcited by 6
  12. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 5
  13. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  14. 2026Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compressioncited by 5
  15. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5
  16. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detectioncited by 4
  17. 2026Evaluating Chain-of-Thought Reasoning through Reusability and Verifiabilitycited by 4
  18. 2025Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creationcited by 3
  19. 2025FaithLens: Detecting and Explaining Faithfulness Hallucinationcited by 3
  20. 2026Position: Evaluation Scores Are Perishable Knowledge Claimscited by 3
  21. 2025KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?cited by 2
  22. 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generationcited by 2
  23. 2026TDA-RC: Task-Driven Alignment for Knowledge-Based Reasoning Chains in Large Language Modelscited by 2
  24. 2026TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunicationscited by 2
  25. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  26. 2026GRACE: Step-Level Benchmark for Faithful Reasoning over Contextcited by 2
  27. 2025TRUST: A Decentralized Framework for Auditing Large Language Model Reasoningcited by 1
  28. 2025Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detectioncited by 1
  29. 2026InFi-Check: Interpretable and Fine-Grained Fact-Checking of LLMscited by 1
  30. 2026LLMs for Game Theory: Entropy-Guided In-Context Learning and Adaptive CoT Reasoningcited by 1
  31. 2026Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScorecited by 1
  32. 2026Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariantscited by 1
  33. 2026Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physicscited by 1
  34. 2026Can LLM Coding Agents Reason About Time Series?cited by 1
  35. 2024Zero-shot Factual Consistency Evaluation Across Domainscited by 0
  36. 2025Generating Privacy Stories From Software Documentationcited by 0
  37. 2025Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning Evaluationcited by 0
  38. 2026ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Modelscited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024General Purpose Verification for Chain of Thought Prompting
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  3. 2024Can We Verify Step by Step for Incorrect Answer Detection?
  4. 2026LCoT-GV: Graph Attention Networks for Verifying Long Reasoning Chains in Large Language Models
  5. 2026Verify to Amplify: Improving Reasoning via Learned Chain-of-Thought Verification
  6. 2026Valid ne Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought
  7. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  8. 2026Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models