mlsanityData to 7 Oct 2026

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

arXiv 2205.10625 ↗First posted 21 May 2022cs.AI cs.CLcited by 1,397

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.

Key facts. Cited by 1,397 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 73 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

32 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  8. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  9. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  10. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  11. 2019Good-Enough Compositional Data Augmentationcited by 158
  12. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
18 more
  1. 2019Neural Logic Machinescited by 144
  2. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  3. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  4. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  5. 2019Compositional generalization through meta sequence-to-sequence learningcited by 98
  6. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  7. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  8. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  9. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  10. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  11. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  12. 2015Learning to Transduce with Unbounded Memorycited by 54
  13. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  14. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51
  15. 2019Compositional Generalization for Primitive Substitutionscited by 48
  16. 2022SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Modelscited by 38
  17. 2015Neural Random-Access Machinescited by 32
  18. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 65; 2023: 345; 2024: 336; 2025: 384; 2026: 267.

1,397 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
38 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  6. 2023Large Language Models as Optimizerscited by 830
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  8. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  13. 2022A Survey on In-context Learningcited by 717
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  15. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  16. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  17. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  18. 2022Large Language Models Can Self-Improvecited by 656
  19. 2022PAL: Program-aided Language Modelscited by 635
  20. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  21. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  22. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  23. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  24. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  25. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  26. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  27. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  28. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  29. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  30. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  31. 2023Augmented Language Models: a Surveycited by 398
  32. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  33. 2023Larger language models do in-context learning differentlycited by 386
  34. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  35. 2023Language Models can Solve Computer Taskscited by 371
  36. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  37. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  38. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338

50 most cited of 1,397 All 1,397 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Generalist Prompting for Large Language Models by Mental Models
  2. 2022Complexity-Based Prompting for Multi-Step Reasoning
  3. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought
  4. 2024Think Beyond Size: Adaptive Prompting for More Effective Reasoning
  5. 2024Break the Chain: Large Language Models Can be Shortcut Reasoners
  6. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
  7. 2026Diverge to Induce Prompting: Multi-Rationale Induction for Zero-Shot Reasoning
  8. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns