mlsanityData to 7 Oct 2026

ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

arXiv 2212.07919 ↗First posted 15 Dec 2022cs.CL cs.LGcited by 175

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 42 of its 54 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

42 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
28 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  9. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  10. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  13. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  14. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  15. 2021Explaining Answers with Entailment Treescited by 148
  16. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  17. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
  18. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  19. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  20. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  21. 2019SciPy 1.0--Fundamental Algorithms for Scientific Computing in Pythoncited by 49
  22. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  23. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  24. 2020Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMRcited by 25
  25. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  26. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  27. 2018A Meaning-based Statistical English Math Word Problem Solvercited by 10
  28. 2019Natural Language Generation Challenges for Explainable AIcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 42; 2024: 38; 2025: 54; 2026: 38.

175 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  6. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  7. 2023Explainability for Large Language Models: A Surveycited by 331
  8. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  11. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  12. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
38 more
  1. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  2. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  3. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  4. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  5. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  6. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 120
  7. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  8. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  9. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  10. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  11. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 102
  12. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  13. 2023Shepherd: A Critic for Language Model Generationcited by 96
  14. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  15. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  16. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  17. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  18. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  19. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  20. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  21. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  22. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  23. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  24. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  25. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  26. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  27. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  28. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  29. 2025LLMs for Explainable AI: A Comprehensive Surveycited by 42
  30. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  31. 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 39
  32. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  33. 2023GRACE: Discriminator-Guided Chain-of-Thought Reasoningcited by 35
  34. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  35. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  36. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  37. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  38. 2024Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuningcited by 30

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Measuring AI Reasoning: A Guide for Researchers
  2. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
  3. 2025JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation
  4. 2024RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners
  5. 2026Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
  6. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
  7. 2025ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
  8. 2026Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness