Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026
Not on the map. Why?
Abstract (arXiv)
Recent advances in Generative Artificial Intelligence (GenAI) have substantially improved the ability of large language models (LLMs) to generate and explain source code. However, their performance on authentic object-oriented programming (OOP) assessments remains insufficiently understood. This study evaluates five widely used GenAI systems, ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and M365 Copilot, using programming tests and examination tasks from an introductory university OOP course. The generated solutions were assessed using the same grading criteria applied to students and compared with historical student results from the same course, as well as findings from the previous year. Common errors were also analyzed to identify recurring limitations across models. All evaluated GenAI systems achieved higher scores than the average student cohort and frequently obtained full marks on longer programming tasks. Nevertheless, they occasionally produced non-compiling code and continued to struggle with advanced OOP concepts, particularly interfaces, abstract classes, and certain inheritance-related tasks. Performance was also limited on graphics-related questions involving image interpretation. Compared with the previous year, the evaluated systems demonstrated noticeable improvements across most assessments while exhibiting several recurring error patterns. The findings provide an updated evaluation of the capabilities and limitations of contemporary GenAI systems on authentic introductory OOP assessments. They also offer evidence that can inform the design of programming assessments, the responsible integration of GenAI tools into software engineering education, and future studies evaluating the evolution of AI-assisted programming.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
17 of 45 references.
References · 17
- 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 79
- 2023Computing Education in the Era of Generative AIcited by 52
- 2023On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?cited by 48
- 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
- 2023ChatGPT Participates in a Computer Science Examcited by 19
- 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
- 2023ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questionscited by 13
- 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
- 2023ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Coursecited by 12
- 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problemscited by 11
- 2023From Large Language Models to Databases and Back: A discussion on research and educationcited by 9
- 2024OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Modelscited by 8
5 more
- 2024LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignmentscited by 6
- 2024Early Adoption of Generative Artificial Intelligence in Computing Education: Emergent Student Use Cases and Perspectives in 2023cited by 5
- 2023Performance of Large Language Models in a Computer Science Degree Programcited by 4
- 2025Computer Science Education in the Age of Generative AIcited by 2
- 2025Student's Use of Generative AI as a Support Tool in an Advanced Web Development Coursecited by 1
Built on by
Not cited yet.