# ChatGPT Participates in a Computer Science Exam

arXiv: https://arxiv.org/abs/2303.09461 · HTML: https://mlsanity.com/paper/2303.09461/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [ChatGPT: A Meta-Analysis after 2.5 Months](https://mlsanity.com/paper/2302.13795/) (2023)
- [The Chess Transformer: Mastering Play using Generative Language Models](https://mlsanity.com/paper/2008.04057/) (2020)

## Built on by (most cited first, 19 of 19)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization](https://mlsanity.com/paper/2405.17067/) (2024)
- [Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice](https://mlsanity.com/paper/2304.12198/) (2023)
- [Comparing large language models and human programmers for generating programming code](https://mlsanity.com/paper/2403.00894/) (2024)
- [Performance of Large Language Models in a Computer Science Degree Program](https://mlsanity.com/paper/2308.02432/) (2023)
- [Detecting The Corruption Of Online Questionnaires By Artificial Intelligence](https://mlsanity.com/paper/2308.07499/) (2023)
- [Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality Assurance](https://mlsanity.com/paper/2401.07441/) (2024)
- [A Roadmap Towards Automated and Regulated Robotic Systems](https://mlsanity.com/paper/2403.14049/) (2024)
- [Exploring the Efficacy of ChatGPT in Analyzing Student Teamwork Feedback with an Existing Taxonomy](https://mlsanity.com/paper/2305.11882/) (2023)
- [Student Mastery or AI Deception? Analyzing ChatGPT's Assessment Proficiency and Evaluating Detection Strategies](https://mlsanity.com/paper/2311.16292/) (2023)
- [Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research](https://mlsanity.com/paper/2312.03497/) (2023)
- [ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Exams](https://mlsanity.com/paper/2309.15866/) (2023)
- [Revisiting The Classics: A Study on Identifying and Rectifying Gender Stereotypes in Rhymes and Poems](https://mlsanity.com/paper/2403.11752/) (2024)
- [PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback](https://mlsanity.com/paper/2502.18425/) (2025)
- [Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026](https://mlsanity.com/paper/2608.16318/) (2026)

## Abstract (arXiv)

We asked ChatGPT to participate in an undergraduate computer science exam on ''Algorithms and Data Structures''. The program was evaluated on the entire exam as posed to the students. We hand-copied its answers onto an exam sheet, which was subsequently graded in a blind setup alongside those of 200 participating students. We find that ChatGPT narrowly passed the exam, obtaining 20.5 out of 40 points. This impressive performance indicates that ChatGPT can indeed succeed in challenging tasks like university exams. At the same time, the questions in our exam are structurally similar to those of other exams, solved homework problems, and teaching materials that can be found online and might have been part of ChatGPT's training data. Therefore, it would be inadequate to conclude from this experiment that ChatGPT has any understanding of computer science. We also assess the improvements brought by GPT-4. We find that GPT-4 would have obtained about 17% more exam points than GPT-3.5, reaching the performance of the average student. The transcripts of our conversations with ChatGPT are available at urlhttps://github.com/tml-tuebingen/chatgpt-algorithm-exam, and the entire graded exam is in the appendix of this paper.