# Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science?

arXiv: https://arxiv.org/abs/2401.15081 · HTML: https://mlsanity.com/paper/2401.15081/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 58 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Builds on (most cited first, 5 of 5)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [NERIF: GPT-4V for Automatic Scoring of Drawn Models](https://mlsanity.com/paper/2311.12990/) (2023)
- [AGI: Artificial General Intelligence for Education](https://mlsanity.com/paper/2304.12479/) (2023)
- [Elucidating STEM Concepts through Generative AI: A Multi-modal Exploration of Analogical Reasoning](https://mlsanity.com/paper/2308.10454/) (2023)

## Built on by (most cited first, 7 of 7)

- [Transforming Teachers' Roles and Agencies in the Era of Generative AI: Perceptions, Acceptance, Knowledge, and Practices](https://mlsanity.com/paper/2410.03018/) (2024)
- [LLMs and Childhood Safety: Identifying Risks and Proposing a Protection Framework for Safe Child-LLM Interaction](https://mlsanity.com/paper/2502.11242/) (2025)
- [Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings](https://mlsanity.com/paper/2503.06980/) (2025)
- [Generative AI and Power Imbalances in Global Education: Frameworks for Bias Mitigation](https://mlsanity.com/paper/2406.02966/) (2024)
- [Assessing AI-Generated vs. Human-Authored Spear Phishing SMS Attacks: An Empirical Study](https://mlsanity.com/paper/2406.13049/) (2024)
- [Ensemble Bayesian Inference: Leveraging Small Language Models to Achieve LLM-level Accuracy in Profile Matching Tasks](https://mlsanity.com/paper/2504.17685/) (2025)
- [Generativism: Toward a Learning Theory for the Age of Generative Artificial Intelligence](https://mlsanity.com/paper/2606.12441/) (2026)

## Abstract (arXiv)

This study aimed to examine an assumption that generative artificial intelligence (GAI) tools can overcome the cognitive intensity that humans suffer when solving problems. We compared the performance of ChatGPT and GPT-4 on 2019 NAEP science assessments with students by cognitive demands of the items. Fifty-four tasks were coded by experts using a two-dimensional cognitive load framework, including task cognitive complexity and dimensionality. ChatGPT and GPT-4 responses were scored using the scoring keys of NAEP. The analysis of the available data was based on the average student ability scores for students who answered each item correctly and the percentage of students who responded to individual items. Results showed that both ChatGPT and GPT-4 consistently outperformed most students who answered the NAEP science assessments. As the cognitive demand for NAEP tasks increases, statistically higher average student ability scores are required to correctly address the questions. This pattern was observed for students in grades 4, 8, and 12, respectively. However, ChatGPT and GPT-4 were not statistically sensitive to the increase in cognitive demands of the tasks, except for Grade 4. As the first study focusing on comparing GAI and K-12 students in problem-solving in science, this finding implies the need for changes to educational objectives to prepare students with competence to work with GAI tools in the future. Education ought to emphasize the cultivation of advanced cognitive skills rather than depending solely on tasks that demand cognitive intensity. This approach would foster critical thinking, analytical skills, and the application of knowledge in novel contexts. Findings also suggest the need for innovative assessment practices by moving away from cognitive intensity tasks toward creativity and analytical skills to avoid the negative effects of GAI on testing more efficiently.