mlsanityData to 7 Oct 2026

Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

arXiv 2310.06117 ↗First posted 9 Oct 2023cs.LG cs.AI cs.CLcited by 157

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide reasoning, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial performance gains on various challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU (Physics and Chemistry) by 7% and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 43 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

32 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
17 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2023Large Language Models as Optimizerscited by 830
  8. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  11. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  12. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  13. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  14. 2022Recitation-Augmented Language Modelscited by 65
  15. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  16. 2022Is a Question Decomposition Unit All We Need?cited by 37
  17. 2021Draw Me a Flower: Processing and Grounding Abstraction in Natural Languagecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 68; 2025: 57; 2026: 27.

157 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  4. 2024Automated Design of Agentic Systemscited by 360
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  7. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  8. 2023An LLM Compiler for Parallel Function Callingcited by 170
  9. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
  10. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  11. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 108
  12. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
38 more
  1. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  2. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  3. 2024CogBench: a large language model walks into a psychology labcited by 60
  4. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  5. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Sciencecited by 56
  6. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  7. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  8. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  9. 2024LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planningcited by 44
  10. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 41
  11. 2024Anchoring Bias in Large Language Models: An Experimental Studycited by 37
  12. 2025TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generationcited by 37
  13. 2025Self-Supervised Prompt Optimizationcited by 37
  14. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  15. 2024Meta Reasoning for Large Language Modelscited by 29
  16. 2024REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answeringcited by 28
  17. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 27
  18. 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAGcited by 26
  19. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 26
  20. 2024Instance-adaptive Zero-shot Chain-of-Thought Promptingcited by 25
  21. 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 24
  22. 2024Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Basescited by 23
  23. 2025GraphRAG under Firecited by 23
  24. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  25. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 22
  26. 2024A Survey on the Honesty of Large Language Modelscited by 21
  27. 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Roomcited by 21
  28. 2024Prompting Fairness: Integrating Causality to Debias Large Language Modelscited by 20
  29. 2024Rational Metareasoning for Large Language Modelscited by 20
  30. 2024MetaReflection: Learning Instructions for Language Agents using Past Reflectionscited by 19
  31. 2024Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitationcited by 19
  32. 2023Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responsescited by 18
  33. 2025PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generationcited by 18
  34. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  35. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 17
  36. 2025Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractionscited by 17
  37. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
  38. 2025Graph RAG-Tool Fusioncited by 15

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversion
  2. 2023Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models
  3. 2023ART: Automatic multi-step reasoning and tool-use for large language models
  4. 2024Think Beyond Size: Adaptive Prompting for More Effective Reasoning
  5. 2025Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
  6. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  7. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models
  8. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models