# How critically can an AI think? A framework for evaluating the quality of thinking of generative artificial intelligence

arXiv: https://arxiv.org/abs/2406.14769 · HTML: https://mlsanity.com/paper/2406.14769/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 10 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 1 of 1)

- [Assigning AI: Seven Approaches for Students, with Prompts](https://mlsanity.com/paper/2306.10052/) (2023)

## Built on by (most cited first, 3 of 3)

- [The Imitation Game for Educational AI](https://mlsanity.com/paper/2502.15127/) (2025)
- [Machine vs Machine: Using AI to Tackle Generative AI Threats in Assessment](https://mlsanity.com/paper/2506.02046/) (2025)
- [Stan: An LLM-based thermodynamics course assistant](https://mlsanity.com/paper/2603.04657/) (2026)

## Abstract (arXiv)

Generative AI such as those with large language models have created opportunities for innovative assessment design practices. Due to recent technological developments, there is a need to know the limits and capabilities of generative AI in terms of simulating cognitive skills. Assessing student critical thinking skills has been a feature of assessment for time immemorial, but the demands of digital assessment create unique challenges for equity, academic integrity and assessment authorship. Educators need a framework for determining their assessments vulnerability to generative AI to inform assessment design practices. This paper presents a framework that explores the capabilities of the LLM ChatGPT4 application, which is the current industry benchmark. This paper presents the Mapping of questions, AI vulnerability testing, Grading, Evaluation (MAGE) framework to methodically critique their assessments within their own disciplinary contexts. This critique will provide specific and targeted indications of their questions vulnerabilities in terms of the critical thinking skills. This can go on to form the basis of assessment design for their tasks.