mlsanityData to 14 Oct 2026

Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use

arXiv 2609.15624 ↗First posted 14 Sep 2026cs.HC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance. We conducted a structured, seeded review of 24 focal empirical publications, starting from the 2024 COSMIN-based review and adding a targeted update through 17 August 2026. We grouped the measures into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. In an exploratory meta-analysis, we pooled three direct subjective-objective correlations from one research program (REML r = .055; Hartung-Knapp 95% CI [-.047, .156]; combined reported N = 2,765). We could not resolve a discrepancy between the largest study's reported correlation and p-value, leaving its weight uncertain. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gave r = .079 (95% CI [-.025, .181]). This sensitivity analysis concerns a broader comparison. From this small evidence base, we cannot establish a population correlation, validate workplace cutoffs, or justify substituting self-ratings for performance scores. We identified tests of foundation knowledge (AICOS-S and GLAT) and measures of verification, reliance, trust, and dependency. We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure; some cover subsets. We propose a four-layer workplace battery with non-compensatory decision rules, but have not tested its thresholds or whether it improves on other assessment approaches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 43 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

13 of 43 references.

References · 13

  1. 2025Authenticated Delegation and Authorized AI Agentscited by 65
  2. 2026Intelligent AI Delegationcited by 18
  3. 2023MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competenciescited by 17
  4. 2024GLAT: The Generative AI Literacy Assessment Testcited by 10
  5. 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
  6. 2025Objective Measurement of AI Literacy: Development and Validation of the AI Competency Objective Scale (AICOS)cited by 4
  7. 2025The AI Literacy Heptagon: A Structured Approach to AI Literacy in Higher Educationcited by 3
  8. 2026How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measurescited by 2
  9. 2015Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studiescited by 1
  10. 2025AI & Data Competencies: Scaffolding holistic AI literacy in Higher Educationcited by 1
  11. 2025AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupationscited by 1
  12. 2025Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scalecited by 1
1 more
  1. 2026Development and Validation of a Faculty Artificial Intelligence Literacy and Competency (FALCON-AI) Scale for Higher Educationcited by 1

Built on by

Not cited yet.