# Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence

arXiv: https://arxiv.org/abs/2312.01109 · HTML: https://mlsanity.com/paper/2312.01109/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 2 Dec 2023.

## Builds on (most cited first, 9 of 9)

- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments](https://mlsanity.com/paper/2308.08572/) (2023)
- [Can ChatGPT Pass An Introductory Level Functional Language Programming Course?](https://mlsanity.com/paper/2305.02230/) (2023)
- [Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Education](https://mlsanity.com/paper/2305.13114/) (2023)
- [Academic Source Code Plagiarism Detection by Measuring Program Behavioural Similarity](https://mlsanity.com/paper/2102.03995/) (2021)

## Built on by (most cited first, 4 of 4)

- [The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course](https://mlsanity.com/paper/2503.07928/) (2025)
- [Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance](https://mlsanity.com/paper/2506.18501/) (2025)
- [Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges](https://mlsanity.com/paper/2509.15283/) (2025)
- [Pedagogical Promise and Peril of AI: A Text Mining Analysis of ChatGPT Research Discussions in Programming Education](https://mlsanity.com/paper/2605.00361/) (2026)

## Abstract (arXiv)

AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPT's ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education.