mlsanityData to 14 Oct 2026

Complexity-Based Prompting for Multi-Step Reasoning

arXiv 2210.00720 ↗First posted 3 Oct 2022cs.CL cs.AI cs.LGcited by 388

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We study the task of prompting large-scale language models to perform multi-step reasoning. Existing work shows that when prompted with a chain of thoughts (CoT), sequences of short sentences describing intermediate reasoning steps towards a final answer, large language models can generate new reasoning chains and predict answers for new inputs. A central question is which reasoning examples make the most effective prompts. In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning. We show that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, achieve substantially better performance on multi-step reasoning tasks over strong baselines. We further extend our complexity-based criteria from prompting (selecting inputs) to decoding (selecting outputs), where we sample multiple reasoning chains from the model, then choose the majority of generated answers from complex reasoning chains (over simple chains). When used to prompt GPT-3 and Codex, our approach substantially improves multi-step reasoning accuracy and achieves new state-of-the-art (SOTA) performance on three math benchmarks (GSM8K, MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared with existing example selection schemes like manual tuning or retrieval-based selection, selection based on reasoning complexity is intuitive, easy to implement, and annotation-efficient. Further results demonstrate the robustness of performance gains from complex prompts under format perturbation and distribution shift.

Key facts. Cited by 388 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 41 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

35 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
20 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  5. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2016Solving General Arithmetic Word Problemscited by 360
  9. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  10. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  11. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  12. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  13. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  14. 2018Did the Model Understand the Question?cited by 60
  15. 2018What Makes Reading Comprehension Questions Easier?cited by 60
  16. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  17. 2018StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsingcited by 22
  18. 2016Learning Executable Semantic Parsers for Natural Language Understandingcited by 21
  19. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18
  20. 2017Learning an Executable Neural Semantic Parsercited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 125; 2024: 126; 2025: 89; 2026: 41.

388 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2025s1: Simple test-time scalingcited by 1,341
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  10. 2022A Survey on In-context Learningcited by 721
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
38 more
  1. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  4. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  5. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  6. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  9. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  10. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  12. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251
  13. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  14. 2023Do Large Language Models Know What They Don't Know?cited by 243
  15. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  16. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 238
  17. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  18. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  19. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  20. 2024More Agents Is All You Needcited by 180
  21. 2025Multi-agent Architecture Search via Agentic Supernetcited by 179
  22. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 178
  23. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  24. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  25. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  26. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  27. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  28. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  29. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  30. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 153
  31. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  32. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  33. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  34. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  35. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  36. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  37. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  38. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128

50 most cited of 388 All 388 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Think Beyond Size: Adaptive Prompting for More Effective Reasoning
  2. 2023Large Language Models as Analogical Reasoners
  3. 2022Automatic Chain of Thought Prompting in Large Language Models
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
  5. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  6. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  7. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
  8. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting