# Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

arXiv: https://arxiv.org/abs/2505.13763 · HTML: https://mlsanity.com/paper/2505.13763/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [The mechanistic basis of data dependence and abrupt learning in an in-context classification task](https://mlsanity.com/paper/2312.03002/) (2023)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples](https://mlsanity.com/paper/2404.07544/) (2024)
- [Obfuscated Activations Bypass LLM Latent-Space Defenses](https://mlsanity.com/paper/2412.09565/) (2024)
- [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025)
- [Metacognitive Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2402.11626/) (2024)
- [Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update](https://mlsanity.com/paper/2501.16378/) (2025)

## Built on by (most cited first, 30 of 41)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Evidence for Limited Metacognition in LLMs](https://mlsanity.com/paper/2509.21545/) (2025)
- [Can LLMs Introspect? A Reality Check](https://mlsanity.com/paper/2605.26242/) (2026)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors](https://mlsanity.com/paper/2512.11949/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use](https://mlsanity.com/paper/2605.14038/) (2026)
- [TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints](https://mlsanity.com/paper/2605.13414/) (2026)
- [Adapting Like Humans: A Metacognitive Agent with Test-time Reasoning](https://mlsanity.com/paper/2511.23262/) (2025)
- [Monitor-Generate-Verify (MGV): Formalising Metacognitive Theory for Language Model Reasoning](https://mlsanity.com/paper/2511.04341/) (2025)
- [Detecting the Disturbance: A Nuanced View of Introspective Abilities in LLMs](https://mlsanity.com/paper/2512.12411/) (2025)
- [Before you <think>, monitor: Implementing Flavell's metacognitive framework in LLMs](https://mlsanity.com/paper/2510.16374/) (2025)
- [Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Models](https://mlsanity.com/paper/2602.02467/) (2026)
- [Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models](https://mlsanity.com/paper/2602.12996/) (2026)
- [Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness](https://mlsanity.com/paper/2604.12373/) (2026)
- [Fine-Tuning Language Models to Know What They Know](https://mlsanity.com/paper/2602.02605/) (2026)
- [Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models](https://mlsanity.com/paper/2604.25922/) (2026)
- [Large language models reorganize representational geometry during in-context learning](https://mlsanity.com/paper/2605.28854/) (2026)
- [Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvement](https://mlsanity.com/paper/2607.04277/) (2026)
- [Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation](https://mlsanity.com/paper/2603.18893/) (2026)
- [Strategic Polysemy in AI Discourse: A Philosophical Analysis of Language, Hype, and Power](https://mlsanity.com/paper/2604.21043/) (2026)
- [Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment](https://mlsanity.com/paper/2606.23700/) (2026)
- [Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates](https://mlsanity.com/paper/2607.01047/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [Diagnosing Correctness Probes under Self-Judgement Confounding](https://mlsanity.com/paper/2607.16799/) (2026)
- [LLMs Learn to Evade Latent Monitors from Prior Feedback Alone](https://mlsanity.com/paper/2609.36490/) (2026)
- [Automatic Minds: Cognitive Parallels Between Hypnotic States and Large Language Model Processing](https://mlsanity.com/paper/2511.01363/) (2025)
- [Stress-Testing Alignment Audits With Prompt-Level Strategic Deception](https://mlsanity.com/paper/2602.08877/) (2026)
- [Confidence Freeze: Early Success Induces a Metastable Decoupling of Metacognition and Behaviour](https://mlsanity.com/paper/2603.21043/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition - the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society's increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired neurofeedback paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to report and control their activation patterns. We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a "metacognitive space" with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).