mlsanityData to 7 Oct 2026

Towards Reasoning in Large Language Models: A Survey

arXiv 2212.10403 ↗First posted 20 Dec 2022cs.CL cs.AIcited by 593

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Reasoning is a fundamental aspect of human intelligence that plays a crucial role in activities such as problem solving, decision making, and critical thinking. In recent years, large language models (LLMs) have made significant progress in natural language processing, and there is observation that these models may exhibit reasoning abilities when they are sufficiently large. However, it is not yet clear to what extent LLMs are capable of reasoning. This paper provides a comprehensive overview of the current state of knowledge on reasoning in LLMs, including techniques for improving and eliciting reasoning in these models, methods and benchmarks for evaluating reasoning abilities, findings and implications of previous research in this field, and suggestions on future directions. Our aim is to provide a detailed and up-to-date review of this topic and stimulate meaningful discussion and future work.

Key facts. Cited by 593 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 91 of its 132 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

91 of 132 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022ALERT: Adapting Language Models to Reasoning Tasks16 Dec
  2. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning15 Dec
  3. 2022PAL: Program-aided Language Models18 Nov
  4. 2022Language Models of Code are Few-Shot Commonsense Learners13 Oct
  5. 2022Explanations from Large Language Models Make Small Reasoners Better13 Oct
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct
  7. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  9. 2022Faithful Reasoning Using Large Language Models30 Aug
  10. 2022Language models show human-like content effects on reasoning tasks14 Jul
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May
  12. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning6 May
2 more
  1. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoning15 Feb
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 77

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  16. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  17. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  18. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  19. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  20. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  22. 2022Galactica: A Large Language Model for Sciencecited by 736
  23. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  24. 2022Large Language Models Can Self-Improvecited by 656
  25. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  26. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  27. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  28. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  29. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  30. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  31. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  32. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  33. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  34. 2016Solving General Arithmetic Word Problemscited by 353
  35. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  36. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  37. 2022Teaching Small Language Models to Reasoncited by 279
  38. 2022Reasoning with Language Model Prompting: A Surveycited by 251

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 139; 2024: 192; 2025: 200; 2026: 62.

593 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2023A Comprehensive Overview of Large Language Modelscited by 639
  9. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  10. 2024Large Language Models: A Surveycited by 506
  11. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  12. 2023Augmented Language Models: a Surveycited by 398
38 more
  1. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  2. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  3. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  4. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
  7. 2023A Survey on Model Compression for Large Language Modelscited by 263
  8. 2023Challenges and Applications of Large Language Modelscited by 251
  9. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  10. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  11. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  12. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  13. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  14. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  15. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  16. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  17. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  18. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  19. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  20. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  21. 2023Large Language Models for Robotics: A Surveycited by 132
  22. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  23. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  24. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  25. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  26. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  27. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  28. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  29. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  30. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  31. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 105
  32. 2023How to Index Item IDs for Recommendation Foundation Modelscited by 102
  33. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  34. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  35. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  36. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  37. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  38. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87

50 most cited of 593 All 593 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  2. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  3. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
  4. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  5. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  6. 2025Reasoning Capabilities and Invariability of Large Language Models
  7. 2025Logical Reasoning in Large Language Models: A Survey
  8. 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Review