# Humans are Still Better than ChatGPT: Case of the IEEEXtreme Competition

arXiv: https://arxiv.org/abs/2305.06934 · HTML: https://mlsanity.com/paper/2305.06934/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 18 of 18)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [ChatGPT for Programming Numerical Methods](https://mlsanity.com/paper/2303.12093/) (2023)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [Investigating the use of ChatGPT for the scheduling of construction projects](https://mlsanity.com/paper/2302.02805/) (2023)
- [Linguistic ambiguity analysis in ChatGPT](https://mlsanity.com/paper/2302.06426/) (2023)
- [Many bioinformatics programming tasks can be automated with ChatGPT](https://mlsanity.com/paper/2303.13528/) (2023)
- [Conversational AI-Powered Design: ChatGPT as Designer, User, and Product](https://mlsanity.com/paper/2302.07406/) (2023)
- [Exploring the Use of ChatGPT as a Tool for Learning and Assessment in Undergraduate Computer Science Curriculum: Opportunities and Challenges](https://mlsanity.com/paper/2304.11214/) (2023)

## Built on by (most cited first, 5 of 5)

- [Large Language Models for Mathematicians](https://mlsanity.com/paper/2312.04556/) (2023)
- [The Path Ahead for Agentic AI: Challenges and Opportunities](https://mlsanity.com/paper/2601.02749/) (2026)
- [InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction](https://mlsanity.com/paper/2305.14659/) (2023)
- [ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Exams](https://mlsanity.com/paper/2309.15866/) (2023)
- [Evolution-Bootstrapped Simulation: Artificial or Human Intelligence: Which Came First?](https://mlsanity.com/paper/2402.00030/) (2024)

## Abstract (arXiv)

Since the release of ChatGPT, numerous studies have highlighted the remarkable performance of ChatGPT, which often rivals or even surpasses human capabilities in various tasks and domains. However, this paper presents a contrasting perspective by demonstrating an instance where human performance excels in typical tasks suited for ChatGPT, specifically in the domain of computer programming. We utilize the IEEExtreme Challenge competition as a benchmark, a prestigious, annual international programming contest encompassing a wide range of problems with different complexities. To conduct a thorough evaluation, we selected and executed a diverse set of 102 challenges, drawn from five distinct IEEExtreme editions, using three major programming languages: Python, Java, and C++. Our empirical analysis provides evidence that contrary to popular belief, human programmers maintain a competitive edge over ChatGPT in certain aspects of problem-solving within the programming context. In fact, we found that the average score obtained by ChatGPT on the set of IEEExtreme programming problems is 3.9 to 5.8 times lower than the average human score, depending on the programming language. This paper elaborates on these findings, offering critical insights into the limitations and potential areas of improvement for AI-based language models like ChatGPT.