mlsanityData to 7 Oct 2026

Progressive-Hint Prompting Improves Reasoning in Large Language Models

arXiv 2304.09797 ↗First posted 19 Apr 2023cs.CL cs.LGcited by 147

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

The performance of Large Language Models (LLMs) in reasoning tasks depends heavily on prompt design, with Chain-of-Thought (CoT) and self-consistency being critical methods that enhance this ability. However, these methods do not fully exploit the answers generated by the LLM to guide subsequent responses. This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP), that enables automatic multiple interactions between users and LLMs by using previously generated answers as hints to progressively guide toward the correct answers. PHP is orthogonal to CoT and self-consistency, making it easy to combine with state-of-the-art techniques to further improve performance. We conducted extensive and comprehensive experiments on seven benchmarks. The results show that PHP significantly improves accuracy while remaining highly efficient. For instance, with text-davinci-003, we observed a 4.2% improvement on GSM8K with greedy decoding compared to Complex CoT, and a 46.17% reduction in sample paths with self-consistency. With GPT-4 and PHP, we achieve state-of-the-art performances on SVAMP (89.1% -> 91.9%), GSM8K (92% -> 95.5%), AQuA (76.4% -> 79.9%) and MATH (50.3% -> 53.9%).

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 70 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

58 of 70 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Boosted Prompt Ensembles for Large Language Models12 Apr
  2. 2023Active Prompting with Chain-of-Thought for Large Language Models23 Feb
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  4. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  5. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  6. 2022Large Language Models are Zero-Shot Reasoners24 May
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
37 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  13. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  14. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  15. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  16. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  17. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  18. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  19. 2016Solving General Arithmetic Word Problemscited by 353
  20. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  21. 2023Mathematical Capabilities of ChatGPTcited by 263
  22. 2022Autoformalization with Large Language Modelscited by 252
  23. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  24. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  25. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  26. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  27. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  28. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  29. 2022Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Proverscited by 97
  30. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  31. 2018What Makes Reading Comprehension Questions Easier?cited by 59
  32. 2019Multi-hop Question Answering via Reasoning Chainscited by 42
  33. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  34. 2022Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Expertscited by 19
  35. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18
  36. 2021Exploiting Reasoning Chains for Multi-hop Science Question Answeringcited by 15
  37. 2017Learning an Executable Neural Semantic Parsercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 45; 2024: 54; 2025: 38; 2026: 10.

147 papers.

Sort:

By area:

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  4. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  5. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  6. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  7. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  8. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  9. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  10. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
38 more
  1. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  2. 2023LEGO-Prover: Neural Theorem Proving with Growing Librariescited by 108
  3. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoningcited by 105
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  6. 2023Cumulative Reasoning with Large Language Modelscited by 94
  7. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 82
  8. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  9. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  10. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  11. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  12. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  13. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  14. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  15. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  16. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  17. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  18. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  19. 2024Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalizationcited by 53
  20. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  21. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  22. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  23. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 48
  24. 2023Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Samplingcited by 46
  25. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  26. 2023Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinementcited by 43
  27. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  28. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  29. 2023MoT: Memory-of-Thought Enables ChatGPT to Self-Improvecited by 37
  30. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  31. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoningcited by 34
  32. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  33. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  34. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  35. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  36. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 31
  37. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  38. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Think Beyond Size: Adaptive Prompting for More Effective Reasoning
  2. 2024Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge
  3. 2026Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs
  4. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  5. 2025Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning
  6. 2022Large Language Models are Better Reasoners with Self-Verification
  7. 2025Asking LLMs to Verify First is Almost Free Lunch
  8. 2023Prompt Sketching for Large Language Models