mlsanityData to 14 Oct 2026

TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

arXiv 2605.13414 ↗First posted 13 May 2026cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

29 of 48 references.

References · 29

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  12. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
17 more
  1. 2024Token-Budget-Aware LLM Reasoningcited by 247
  2. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  3. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 163
  4. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
  5. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  6. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  7. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  8. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 84
  9. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  10. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  11. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 48
  12. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 41
  13. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  14. 2025Metacognition and Uncertainty Communication in Humans and Large Language Modelscited by 23
  15. 2025Evidence for Limited Metacognition in LLMscited by 12
  16. 2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 10
  17. 2026W&D:Scaling Parallel Tool Calling for Efficient Deep Research Agentscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Modelscited by 2
  2. 2026Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questionscited by 1
  3. 2026BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answeringcited by 1
  4. 2026EvalResearchBench: Can AI Agents Design Their Own Evaluations?cited by 1
  5. 2026Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commercecited by 0
  6. 2026New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Modelscited by 0