# Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use

arXiv: https://arxiv.org/abs/2609.15624 · HTML: https://mlsanity.com/paper/2609.15624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Authenticated Delegation and Authorized AI Agents](https://mlsanity.com/paper/2501.09674/) (2025)
- [Intelligent AI Delegation](https://mlsanity.com/paper/2602.11865/) (2026)
- [MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies](https://mlsanity.com/paper/2302.09319/) (2023)
- [GLAT: The Generative AI Literacy Assessment Test](https://mlsanity.com/paper/2411.00283/) (2024)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [Objective Measurement of AI Literacy: Development and Validation of the AI Competency Objective Scale (AICOS)](https://mlsanity.com/paper/2503.12921/) (2025)
- [The AI Literacy Heptagon: A Structured Approach to AI Literacy in Higher Education](https://mlsanity.com/paper/2509.18900/) (2025)
- [How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures](https://mlsanity.com/paper/2601.06101/) (2026)
- [Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies](https://mlsanity.com/paper/1508.01227/) (2015)
- [AI & Data Competencies: Scaffolding holistic AI literacy in Higher Education](https://mlsanity.com/paper/2510.24783/) (2025)
- [AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupations](https://mlsanity.com/paper/2511.05475/) (2025)
- [Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scale](https://mlsanity.com/paper/2512.12413/) (2025)
- [Development and Validation of a Faculty Artificial Intelligence Literacy and Competency (FALCON-AI) Scale for Higher Education](https://mlsanity.com/paper/2603.20220/) (2026)

## Abstract (arXiv)

Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance. We conducted a structured, seeded review of 24 focal empirical publications, starting from the 2024 COSMIN-based review and adding a targeted update through 17 August 2026. We grouped the measures into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. In an exploratory meta-analysis, we pooled three direct subjective-objective correlations from one research program (REML r = .055; Hartung-Knapp 95% CI [-.047, .156]; combined reported N = 2,765). We could not resolve a discrepancy between the largest study's reported correlation and p-value, leaving its weight uncertain. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gave r = .079 (95% CI [-.025, .181]). This sensitivity analysis concerns a broader comparison. From this small evidence base, we cannot establish a population correlation, validate workplace cutoffs, or justify substituting self-ratings for performance scores. We identified tests of foundation knowledge (AICOS-S and GLAT) and measures of verification, reliance, trust, and dependency. We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure; some cover subsets. We propose a four-layer workplace battery with non-compensatory decision rules, but have not tested its thresholds or whether it improves on other assessment approaches.