# Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas

arXiv: https://arxiv.org/abs/2605.06673 · HTML: https://mlsanity.com/paper/2605.06673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [How do LLMs Compute Verbal Confidence](https://mlsanity.com/paper/2603.17839/) (2026)
- [Metacognition and Uncertainty Communication in Humans and Large Language Models](https://mlsanity.com/paper/2504.14045/) (2025)
- [Closing the Confidence-Faithfulness Gap in Large Language Models](https://mlsanity.com/paper/2603.25052/) (2026)
- [LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy](https://mlsanity.com/paper/2603.14893/) (2026)
- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://mlsanity.com/paper/2604.03216/) (2026)
- [A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities](https://mlsanity.com/paper/2603.02540/) (2026)
- [The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring](https://mlsanity.com/paper/2604.15702/) (2026)
- [Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report](https://mlsanity.com/paper/2604.17707/) (2026)
- [Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals](https://mlsanity.com/paper/2604.17714/) (2026)
- [Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction](https://mlsanity.com/paper/2604.17716/) (2026)

## Abstract (arXiv)

Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, under an a priori six-domain grouping) to 33 frontier LLMs from eight model families and computed Type-2 AUROC per model-domain cell using verbalized confidence (0-100). Total observations: 47,151. Every model with above-chance aggregate monitoring showed non-trivial domain-level variation. Applied/Professional knowledge was reliably the easiest benchmark domain to monitor (mean AUROC = .742, ranked top-2 in 21 of 33 models); Formal Reasoning and Natural Science were reliably the hardest (one of the two ranked bottom-2 in 27 of 33 models). The three middle domains were statistically indistinguishable (Kendall's W = .164). A subject-level coherence analysis (within-domain similarity ratio = 0.95) confirms the six-domain grouping is a pragmatic benchmark taxonomy, not a validated latent construct. Within-family profile-shape clustering is significant for Anthropic, Google-Gemini, and Qwen (permutation p < .0001) but not DeepSeek, Google-Gemma, or OpenAI. Gemma 4 31B showed a +.202 AUROC improvement over Gemma 3 27B. Three models classified Invalid on binary KEEP/WITHDRAW probes produced normal profiles under verbalized confidence, confirming probe-format specificity. Bootstrap 95% CIs on 198 cells have median width .199. Split-half aggregate stability r = .893; profile-level split-half is weaker (grand median r = .184). These results show stable benchmark-domain variation obscured by aggregate metrics, and support benchmark-stage domain screening as a step before deployment in specific application areas.