mlsanityData to 14 Oct 2026

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

arXiv 2505.10185 ↗First posted 15 May 2025cs.CL cs.AIcited by 9

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Long chain-of-thought (CoT) is an essential ingredient in effective usage of modern large language models, but our understanding of the reasoning strategies underlying these capabilities remains limited. While some prior works have attempted to categorize CoTs using predefined strategy types, such approaches are constrained by human intuition and fail to capture the full diversity of model behaviors. In this work, we introduce the CoT Encyclopedia, a bottom-up framework for analyzing and steering model reasoning. Our method automatically extracts diverse reasoning criteria from model-generated CoTs, embeds them into a semantic space, clusters them into representative categories, and derives contrastive rubrics to interpret reasoning behavior. Human evaluations show that this framework produces more interpretable and comprehensive analyses than existing methods. Moreover, we demonstrate that this understanding enables performance gains: we can predict which strategy a model is likely to use and guide it toward more effective alternatives. Finally, we provide practical insights, such as that training data format (e.g., free-form vs. multiple-choice) has a far greater impact on reasoning behavior than data domain, underscoring the importance of format-aware model design.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 27 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

25 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning21 Jan
  2. 2023LaRS: Latent Reasoning Skills for Chain-of-Thought Reasoning7 Dec
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 20

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2025s1: Simple test-time scalingcited by 1,341
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  12. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
8 more
  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  3. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  4. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  5. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 84
  6. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  7. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  8. 2025ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
  2. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  3. 2024A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning
  4. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotations
  5. 2024Understanding Chain-of-Thought in LLMs through Information Theory
  6. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents
  7. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoning