mlsanityData to 7 Oct 2026

Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving

arXiv 2405.12205 ↗First posted 20 May 2024cs.AI cs.LG stat.MLcited by 82

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Metacognitive knowledge refers to humans' intuitive knowledge of their own thinking and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also have metacognitive knowledge, including ability to name skills and procedures to apply given a task. We explore this primarily in context of math reasoning, developing a prompt-guided interaction procedure to get a powerful LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans. To validate that these skill labels are meaningful and relevant to the LLM's reasoning processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH. (b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed. Then it is presented with randomly selected exemplar solved questions associated with that skill label. This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic, even though this article applies it to math problems.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 49 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

28 of 49 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Mistral 7Bcited by 3,065
  12. 2023PaLM 2 Technical Reportcited by 1,154
16 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  2. 2023Large Language Models as Optimizerscited by 830
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2022PAL: Program-aided Language Modelscited by 635
  5. 2016Solving General Arithmetic Word Problemscited by 353
  6. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  8. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  9. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  10. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  11. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  12. 2023Cumulative Reasoning with Large Language Modelscited by 94
  13. 2024How to Train Data-Efficient LLMscited by 91
  14. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  15. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  16. 2023Evaluation of large language models for discovery of gene set functioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 42; 2026: 25.

82 papers.

Sort:

By area:

  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  2. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  3. 2025Agentic Large Language Models, a surveycited by 105
  4. 2025AI-Researcher: Autonomous Scientific Innovationcited by 88
  5. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  6. 2026Privileged Information Distillation for Language Modelscited by 79
  7. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  8. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  9. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  10. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  11. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  12. 2024Theorem-Validated Reverse Chain-of-Thought Problem Generation for Geometric Reasoningcited by 32
38 more
  1. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  2. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  3. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  4. 2025Metacognition and Uncertainty Communication in Humans and Large Language Modelscited by 23
  5. 2025Does It Make Sense to Speak of Introspection in Large Language Models?cited by 22
  6. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  7. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  8. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 13
  9. 2024Imagining and building wise machines: The centrality of AI metacognitioncited by 12
  10. 2024Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracingcited by 11
  11. 2025SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answerscited by 11
  12. 2024Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Modelscited by 9
  13. 2025The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Thinkcited by 9
  14. 2025Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystemscited by 9
  15. 2025A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysiscited by 9
  16. 2026Learning to Simulate Human Dialoguecited by 9
  17. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  18. 2025BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systemscited by 8
  19. 2025Evaluation of LLMs for mathematical problem solvingcited by 7
  20. 2025AI Awarenesscited by 6
  21. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  22. 2025DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTScited by 6
  23. 2025Skill-Targeted Adaptive Trainingcited by 6
  24. 2024A Triumvirate of AI Driven Theoretical Discoverycited by 5
  25. 2024Smart Vision-Language Reasonerscited by 5
  26. 2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
  27. 2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 5
  28. 2024When LLMs Learn to be Students: The SOEI Framework for Modeling and Evaluating Virtual Student Agents in Educational Interactioncited by 4
  29. 2025Electronic Circuit Principles of Large Language Modelscited by 4
  30. 2025MAPS: Multi-Agent Personality Shaping for Collaborative Reasoningcited by 4
  31. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 4
  32. 2025Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracingcited by 4
  33. 2025ESCA: Contextualizing Embodied Agents via Scene-Graph Generationcited by 4
  34. 2025Monitor-Generate-Verify (MGV): Formalising Metacognitive Theory for Language Model Reasoningcited by 4
  35. 2026Skill Neologisms: Towards Skill-based Continual Learningcited by 4
  36. 2024Large Language Model as a Catalyst: A Paradigm Shift in Base Station Siting Optimizationcited by 3
  37. 2025LemmaHead: RAG Assisted Proof Generation Using Large Language Modelscited by 3
  38. 2025Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decompositioncited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
  2. 2025SMART: Self-Generating and Self-Validating Multi-Dimensional Assessment for LLMs' Mathematical Problem Solving
  3. 2026Math Reasoning in LLMs is Organized by Approach, Not Topic
  4. 2026Beyond Meta-Reasoning: Metacognitive Consolidation for Self-Improving LLM Reasoning
  5. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  6. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
  7. 2026Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory
  8. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities