mlsanityData to 14 Oct 2026

Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysis

arXiv 2409.09054 ↗First posted 31 Aug 2024cs.SE cs.AI cs.PLcited by 0

Not on the map. Why?

Abstract (arXiv)

This study explores the performance of large language models (LLMs) in solving competitive programming problems from the Romanian Informatics Olympiad at the county level. Romania, a leading nation in computer science competitions, provides an ideal environment for evaluating LLM capabilities due to its rich history and stringent competition standards. We collected and analyzed a dataset comprising 304 challenges from 2002 to 2023, focusing on solutions written by LLMs in C++ and Python for these problems. Our primary goal is to understand why LLMs perform well or poorly on different tasks. We evaluated various models, including closed-source models like GPT-4 and open-weight models such as CodeLlama and RoMistral, using a standardized process involving multiple attempts and feedback rounds. The analysis revealed significant variations in LLM performance across different grades and problem types. Notably, GPT-4 showed strong performance, indicating its potential use as an educational tool for middle school students. We also observed differences in code quality and style across various LLMs

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 31 Aug 2024.

Builds on

21 of 42 references.

References · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Mistral 7Bcited by 3,164
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  10. 2021Measuring Coding Challenge Competence With APPScited by 902
  11. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  12. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 170
9 more
  1. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  2. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 93
  3. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  4. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  5. 2024AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instructcited by 8
  6. 2024OpenLLM-Ro -- Technical Report on Open-source Romanian LLMscited by 7
  7. 2023QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Modelscited by 6
  8. 2023Problem-Solving Guide: Predicting the Algorithm Tags and Difficulty for Competitive Programming Problemscited by 5
  9. 2024RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romaniancited by 3

Built on by

Not cited yet.