mlsanityData to 7 Oct 2026

ChatGPT Participates in a Computer Science Exam

arXiv 2303.09461 ↗First posted 8 Mar 2023cs.CL cs.CYcited by 19

Not on the map. Why?

Abstract (arXiv)

We asked ChatGPT to participate in an undergraduate computer science exam on ''Algorithms and Data Structures''. The program was evaluated on the entire exam as posed to the students. We hand-copied its answers onto an exam sheet, which was subsequently graded in a blind setup alongside those of 200 participating students. We find that ChatGPT narrowly passed the exam, obtaining 20.5 out of 40 points. This impressive performance indicates that ChatGPT can indeed succeed in challenging tasks like university exams. At the same time, the questions in our exam are structurally similar to those of other exams, solved homework problems, and teaching materials that can be found online and might have been part of ChatGPT's training data. Therefore, it would be inadequate to conclude from this experiment that ChatGPT has any understanding of computer science. We also assess the improvements brought by GPT-4. We find that GPT-4 would have obtained about 17% more exam points than GPT-3.5, reaching the performance of the average student. The transcripts of our conversations with ChatGPT are available at urlhttps://github.com/tml-tuebingen/chatgpt-algorithm-exam, and the entire graded exam is in the appendix of this paper.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 8 Mar 2023.

Builds on

17 of 26 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Using cognitive psychology to understand GPT-3cited by 389
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  10. 2023Mathematical Capabilities of ChatGPTcited by 263
  11. 2023A Categorical Archive of ChatGPT Failurescited by 167
  12. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
5 more
  1. 2023Learning Performance-Improving Code Editscited by 103
  2. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  3. 2022GPT Takes the Bar Examcited by 65
  4. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 50
  5. 2020The Chess Transformer: Mastering Play using Generative Language Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 5; 2025: 1; 2026: 1.

19 papers.

Sort:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  4. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  5. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  6. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  7. 2023Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practicecited by 15
  8. 2024Comparing large language models and human programmers for generating programming codecited by 9
  9. 2023Performance of Large Language Models in a Computer Science Degree Programcited by 4
  10. 2023Detecting The Corruption Of Online Questionnaires By Artificial Intelligencecited by 3
  11. 2024Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality Assurancecited by 2
  12. 2024A Roadmap Towards Automated and Regulated Robotic Systemscited by 2
7 more
  1. 2023Exploring the Efficacy of ChatGPT in Analyzing Student Teamwork Feedback with an Existing Taxonomycited by 1
  2. 2023Student Mastery or AI Deception? Analyzing ChatGPT's Assessment Proficiency and Evaluating Detection Strategiescited by 1
  3. 2023Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Researchcited by 1
  4. 2023ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Examscited by 0
  5. 2024Revisiting The Classics: A Study on Identifying and Rectifying Gender Stereotypes in Rhymes and Poemscited by 0
  6. 2025PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedbackcited by 0
  7. 2026Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026cited by 0