mlsanityData to 14 Oct 2026

Using profiles of cognitive capability to assess AI suitability for workplace tasks

arXiv 2608.25623 ↗First posted 26 Aug 2026cs.AI cs.CY cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an agent's capabilities from performance on a benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 88 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

38 of 88 references.

References · 38

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  6. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  7. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  8. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  9. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  10. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  11. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  12. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
26 more
  1. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  2. 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 96
  3. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  4. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  5. 2026Towards a Science of AI Agent Reliabilitycited by 65
  6. 2025Artificial Intelligence Index Report 2025cited by 64
  7. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  8. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  9. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  10. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  11. 2025Measuring Agents in Productioncited by 37
  12. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 34
  13. 2025General Scales Unlock AI Evaluation with Explanatory and Predictive Powercited by 31
  14. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  15. 2024Measuring Social Norms of Large Language Modelscited by 17
  16. 2025Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Modelscited by 14
  17. 2023Inferring Capabilities from Task Performance with Bayesian Triangulationcited by 13
  18. 2025The Illusion of Readiness in Health AIcited by 12
  19. 2023Predictable Artificial Intelligencecited by 8
  20. 2026Capabilities Ain't All You Need: Measuring Propensities in AIcited by 6
  21. 2026Measuring Progress Toward AGI: A Cognitive Frameworkcited by 6
  22. 2026Cognitive Dark Matter: Measuring What AI Missescited by 5
  23. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 3
  24. 2023The Animal-AI Environment: A Virtual Laboratory For Comparative Cognition and Artificial Intelligence Researchcited by 2
  25. 2026Visuospatial Perspective Taking in Multimodal Language Modelscited by 2
  26. 2026From Human-Level AI Tales to AI Leveling Human Scalescited by 1

Built on by

Not cited yet.