# Metacognitive Sensitivity for Test-Time Dynamic Model Selection

arXiv: https://arxiv.org/abs/2512.10451 · HTML: https://mlsanity.com/paper/2512.10451/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Builds on (most cited first, 24 of 24)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts](https://mlsanity.com/paper/2405.11273/) (2024)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Self-supervised Domain Adaptation for Computer Vision Tasks](https://mlsanity.com/paper/1907.10915/) (2019)
- [CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://mlsanity.com/paper/2405.05949/) (2024)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [Metacognition and Uncertainty Communication in Humans and Large Language Models](https://mlsanity.com/paper/2504.14045/) (2025)
- [How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech](https://mlsanity.com/paper/2301.11462/) (2023)
- [Privileged Self-Access Matters for Introspection in AI](https://mlsanity.com/paper/2508.14802/) (2025)
- [In-Context Learning Dynamics with Random Binary Sequences](https://mlsanity.com/paper/2310.17639/) (2023)
- [Human-like Affective Cognition in Foundation Models](https://mlsanity.com/paper/2409.11733/) (2024)
- [Large (Vision) Language Models are Unsupervised In-Context Learners](https://mlsanity.com/paper/2504.02349/) (2025)
- [META-DES.H: a dynamic ensemble selection technique using meta-learning and a dynamic weighting approach](https://mlsanity.com/paper/1811.01742/) (2018)
- [Joint Decision-Making in Robot Teleoperation: When are Two Heads Better Than One?](https://mlsanity.com/paper/2503.15510/) (2025)
- [Divide, Specialize, and Route: A New Approach to Efficient Ensemble Learning](https://mlsanity.com/paper/2506.20814/) (2025)

## Built on by (most cited first, 2 of 2)

- [Measuring (some aspects of) the metacognition of AI](https://mlsanity.com/paper/2603.29693/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)

## Abstract (arXiv)

A key aspect of human cognition is metacognition - the ability to assess one's own knowledge and judgment reliability. While deep learning models can express confidence in their predictions, they often suffer from poor calibration, a cognitive bias where expressed confidence does not reflect true competence. Do models truly know what they know? Drawing from human cognitive science, we propose a new framework for evaluating and leveraging AI metacognition. We introduce meta-d', a psychologically-grounded measure of metacognitive sensitivity, to characterise how reliably a model's confidence predicts its own accuracy. We then use this dynamic sensitivity score as context for a bandit-based arbiter that performs test-time model selection, learning which of several expert models to trust for a given task. Our experiments across multiple datasets and deep learning model combinations (including CNNs and VLMs) demonstrate that this metacognitive approach improves joint-inference accuracy over constituent models. This work provides a novel behavioural account of AI models, recasting ensemble selection as a problem of evaluating both short-term signals (confidence prediction scores) and medium-term traits (metacognitive sensitivity).