# Imagining and building wise machines: The centrality of AI metacognition

arXiv: https://arxiv.org/abs/2411.02478 · HTML: https://mlsanity.com/paper/2411.02478/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 131 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving](https://mlsanity.com/paper/2405.12205/) (2024)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Metacognitive Prompting Improves Understanding in Large Language Models](https://mlsanity.com/paper/2308.05342/) (2023)
- [People construct simplified mental representations to plan](https://mlsanity.com/paper/2105.06948/) (2021)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [Metacognitive Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2402.11626/) (2024)
- [I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench](https://mlsanity.com/paper/2401.17882/) (2024)
- [Reasoning About Beliefs and Actions Under Computational Resource Constraints](https://mlsanity.com/paper/1304.2759/) (2013)
- [Towards a Quantum-Like Cognitive Architecture for Decision-Making](https://mlsanity.com/paper/1905.05176/) (2019)
- [Metacognition is all you need? Using Introspection in Generative Agents to Improve Goal-directed Behavior](https://mlsanity.com/paper/2401.10910/) (2024)
- [Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approach](https://mlsanity.com/paper/2403.05636/) (2024)
- [OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond](https://mlsanity.com/paper/2310.18318/) (2023)
- [Can ChatGPT Make Explanatory Inferences? Benchmarks for Abductive Reasoning](https://mlsanity.com/paper/2404.18982/) (2024)
- [Metacognitive Myopia in Large Language Models](https://mlsanity.com/paper/2408.05568/) (2024)
- [Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance](https://mlsanity.com/paper/2405.05285/) (2024)

## Built on by (most cited first, 12 of 12)

- [Evaluating the Robustness of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2411.14215/) (2024)
- [Agentic AI Needs a Systems Theory](https://mlsanity.com/paper/2503.00237/) (2025)
- [Truly Self-Improving Agents Require Intrinsic Metacognitive Learning](https://mlsanity.com/paper/2506.05109/) (2025)
- [Evaluating Long-Term Memory for Long-Context Question Answering](https://mlsanity.com/paper/2510.23730/) (2025)
- [Do Chains-of-Thoughts of Large Language Models Suffer from Hallucinations, Cognitive Biases, or Phobias in Bayesian Reasoning?](https://mlsanity.com/paper/2503.15268/) (2025)
- [Consistency-based Abductive Reasoning over Perceptual Errors of Multiple Pre-trained Models in Novel Environments](https://mlsanity.com/paper/2505.19361/) (2025)
- [Contemplative Artificial Intelligence](https://mlsanity.com/paper/2504.15125/) (2025)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [Realist and Pluralist Conceptions of Intelligence and Their Implications on AI Research](https://mlsanity.com/paper/2511.15282/) (2025)
- [Position: Artificial Intelligence Needs Meta Intelligence -- the Case for Metacognitive AI](https://mlsanity.com/paper/2605.15567/) (2026)
- [The Biomimetic Architecture of Software 4.0](https://mlsanity.com/paper/2606.04025/) (2026)
- [Psychological Competence as a Missing Dimension in AI Evaluation](https://mlsanity.com/paper/2607.08285/) (2026)

## Abstract (arXiv)

Although AI has become increasingly smart, its wisdom has not kept pace. In this article, we examine what is known about human wisdom and sketch a vision of its AI counterpart. We analyze human wisdom as a set of strategies for solving intractable problems-those outside the scope of analytic techniques-including both object-level strategies like heuristics [for managing problems] and metacognitive strategies like intellectual humility, perspective-taking, or context-adaptability [for managing object-level strategies]. We argue that AI systems particularly struggle with metacognition; improved metacognition would lead to AI more robust to novel environments, explainable to users, cooperative with others, and safer in risking fewer misaligned goals with human users. We discuss how wise AI might be benchmarked, trained, and implemented.