# Evidence for Limited Metacognition in LLMs

arXiv: https://arxiv.org/abs/2509.21545 · HTML: https://mlsanity.com/paper/2509.21545/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Consciousness in Artificial Intelligence: Insights from the Science of Consciousness](https://mlsanity.com/paper/2308.08708/) (2023)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024)
- [Could a Large Language Model be Conscious?](https://mlsanity.com/paper/2303.07103/) (2023)
- [Large Language Models Pass the Turing Test](https://mlsanity.com/paper/2503.23674/) (2025)
- [Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations](https://mlsanity.com/paper/2505.13763/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs](https://mlsanity.com/paper/2305.14279/) (2023)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions](https://mlsanity.com/paper/2505.17120/) (2025)
- [Privileged Self-Access Matters for Introspection in AI](https://mlsanity.com/paper/2508.14802/) (2025)
- [Perceptions of Sentient AI and Other Digital Minds: Evidence from the AI, Morality, and Sentience (AIMS) Survey](https://mlsanity.com/paper/2407.08867/) (2024)
- [How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models](https://mlsanity.com/paper/2507.03120/) (2025)
- [Towards a Theory of AI Personhood](https://mlsanity.com/paper/2501.13533/) (2025)

## Built on by (most cited first, 12 of 12)

- [TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints](https://mlsanity.com/paper/2605.13414/) (2026)
- [Hallucinations Undermine Trust; Metacognition is a Way Forward](https://mlsanity.com/paper/2605.01428/) (2026)
- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs](https://mlsanity.com/paper/2606.11063/) (2026)
- [LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception](https://mlsanity.com/paper/2606.30005/) (2026)
- [Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind](https://mlsanity.com/paper/2603.26089/) (2026)
- [Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models](https://mlsanity.com/paper/2604.25922/) (2026)
- [Artificial Phantasia: Emergent Mental Imagery in Large Language Models](https://mlsanity.com/paper/2509.23108/) (2025)
- [When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models](https://mlsanity.com/paper/2606.22974/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [LLMs Learn to Evade Latent Monitors from Prior Feedback Alone](https://mlsanity.com/paper/2609.36490/) (2026)
- [Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory](https://mlsanity.com/paper/2607.23927/) (2026)

## Abstract (arXiv)

The possibility of LLM self-awareness and even sentience is gaining increasing public attention and has major safety and policy implications, but the science of measuring them is still in a nascent state. Here we introduce a novel methodology for quantitatively evaluating metacognitive abilities in LLMs. Taking inspiration from research on metacognition in nonhuman animals, our approach eschews model self-reports and instead tests to what degree models can strategically deploy knowledge of internal states. Using two experimental paradigms, we demonstrate that frontier LLMs introduced since early 2024 show increasingly strong evidence of certain metacognitive abilities, specifically the ability to assess and utilize their own confidence in their ability to answer factual and reasoning questions correctly and the ability to anticipate what answers they would give and utilize that information appropriately. We buttress these behavioral findings with an analysis of the token probabilities returned by the models, which suggests the presence of an upstream internal signal that could provide the basis for metacognition. We further find that these abilities 1) are limited in resolution, 2) emerge in context-dependent manners, and 3) seem to be qualitatively different from those of humans. We also report intriguing differences across models of similar capabilities, suggesting that LLM post-training may have a role in developing metacognitive abilities.