mlsanityData to 7 Oct 2026

Automatic Chain of Thought Prompting in Large Language Models

arXiv 2210.03493 ↗First posted 7 Oct 2022cs.CL cs.AIcited by 684

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the "Let's think step by step" prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot

Key facts. Cited by 684 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 39 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

31 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
15 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2016Solving General Arithmetic Word Problemscited by 353
  11. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  12. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  13. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  14. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  15. 2022Rationale-Augmented Ensembles in Language Modelscited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 188; 2024: 194; 2025: 216; 2026: 76.

684 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2022A Survey on In-context Learningcited by 717
  9. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
38 more
  1. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  2. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  3. 2024Large Language Models: A Surveycited by 506
  4. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  5. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  6. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  7. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  8. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  9. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  10. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  11. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  12. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  13. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  14. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  15. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  16. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  17. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  18. 2023One for All: Towards Training One Graph Model for All Classification Taskscited by 239
  19. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  20. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  21. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  22. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  23. 2024Many-Shot In-Context Learningcited by 203
  24. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  25. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  26. 2023Efficient Large Language Models: A Surveycited by 183
  27. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  28. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  29. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  30. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  31. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  32. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  33. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  34. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  35. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  36. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  37. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  38. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136

50 most cited of 684 All 684 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  2. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  3. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  4. 2024Supervised Chain of Thought
  5. 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
  6. 2024Chain-of-Thought Reasoning Without Prompting
  7. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
  8. 2022Complexity-Based Prompting for Multi-Step Reasoning