# Performance of Large Language Models in a Computer Science Degree Program

arXiv: https://arxiv.org/abs/2308.02432 · HTML: https://mlsanity.com/paper/2308.02432/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://mlsanity.com/paper/2303.08302/) (2023)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [ChatGPT Participates in a Computer Science Exam](https://mlsanity.com/paper/2303.09461/) (2023)

## Built on by (most cited first, 4 of 4)

- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning](https://mlsanity.com/paper/2408.13787/) (2024)
- [Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison](https://mlsanity.com/paper/2507.03013/) (2025)
- [Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026](https://mlsanity.com/paper/2608.16318/) (2026)

## Abstract (arXiv)

Large language models such as ChatGPT-3.5 and GPT-4.0 are ubiquitous and dominate the current discourse. Their transformative capabilities have led to a paradigm shift in how we interact with and utilize (text-based) information. Each day, new possibilities to leverage the capabilities of these models emerge. This paper presents findings on the performance of different large language models in a university of applied sciences' undergraduate computer science degree program. Our primary objective is to assess the effectiveness of these models within the curriculum by employing them as educational aids. By prompting the models with lecture material, exercise tasks, and past exams, we aim to evaluate their proficiency across different computer science domains. We showcase the strong performance of current large language models while highlighting limitations and constraints within the context of such a degree program. We found that ChatGPT-3.5 averaged 79.9% of the total score in 10 tested modules, BingAI achieved 68.4%, and LLaMa, in the 65 billion parameter variant, 20%. Despite these convincing results, even GPT-4.0 would not pass the degree program - due to limitations in mathematical calculations.