# A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course

arXiv: https://arxiv.org/abs/2403.16977 · HTML: https://mlsanity.com/paper/2403.16977/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023)
- [AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?](https://mlsanity.com/paper/2303.01067/) (2023)

## Built on by (most cited first, 10 of 10)

- [BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks](https://mlsanity.com/paper/2411.07464/) (2024)
- [StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis](https://mlsanity.com/paper/2502.17657/) (2025)
- [CueTip: An Interactive and Explainable Physics-aware Pool Assistant](https://mlsanity.com/paper/2501.18291/) (2025)
- [Measuring Hong Kong Massive Multi-Task Language Understanding](https://mlsanity.com/paper/2505.02177/) (2025)
- [Do LLMs produce texts with "human-like" lexical diversity?](https://mlsanity.com/paper/2508.00086/) (2025)
- [Solve it with EASE](https://mlsanity.com/paper/2509.18108/) (2025)
- [From Human-Level AI Tales to AI Leveling Human Scales](https://mlsanity.com/paper/2602.18911/) (2026)
- [LLM-as-a-judge validity is strongly task-dependent across physics assessment formats](https://mlsanity.com/paper/2603.14732/) (2026)
- [Evaluation of GPT-based large language generative AI models as study aids for the national licensure examination for registered dietitians in Japan](https://mlsanity.com/paper/2508.10011/) (2025)
- [Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus](https://mlsanity.com/paper/2510.22318/) (2025)

## Abstract (arXiv)

This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed n = 300 data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = 2.482 × 10-10). Prompt engineering significantly improved scores for both GPT-4 (p = 1.661 × 10-4) and GPT-3.5 (p = 4.967 × 10-9). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from `Definitely AI' to `Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary `AI' or `Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.