mlsanityData to 7 Oct 2026

Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence

arXiv 2312.01109 ↗First posted 2 Dec 2023cs.AI cs.CY cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPT's ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 9 of its 34 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

9 of 34 references.

References · 9

  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 97
  3. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 87
  4. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  5. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  6. 2023Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessmentscited by 10
  7. 2023Can ChatGPT Pass An Introductory Level Functional Language Programming Course?cited by 9
  8. 2023Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Educationcited by 8
  9. 2021Academic Source Code Plagiarism Detection by Measuring Program Behavioural Similaritycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Coursecited by 7
  2. 2025Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performancecited by 1
  3. 2025Evaluating the Limitations of Local LLMs in Solving Complex Programming Challengescited by 0
  4. 2026Pedagogical Promise and Peril of AI: A Text Mining Analysis of ChatGPT Research Discussions in Programming Educationcited by 0