mlsanityData to 14 Oct 2026

Competition-Level Problems are Effective LLM Evaluators

arXiv 2312.02143 ↗First posted 4 Dec 2023cs.CL cs.AIcited by 25

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently. This paper aims to evaluate the reasoning capacities of LLMs, specifically in solving recent competition-level programming problems in Codeforces, which are expert-crafted and unique, requiring deep understanding and robust reasoning skills. We first provide a comprehensive evaluation of GPT-4's peiceived zero-shot performance on this task, considering various aspects such as problems' release time, difficulties, and types of errors encountered. Surprisingly, the peiceived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems, which shows the potential data contamination, as well as the challenges for any existing LLM to solve unseen complex reasoning problems. We further explore various approaches such as fine-tuning, Chain-of-Thought prompting and problem description simplification, unfortunately none of them is able to consistently mitigate the challenges. Through our work, we emphasis the importance of this excellent data source for assessing the genuine reasoning capabilities of LLMs, and foster the development of LLMs with stronger reasoning abilities and better generalization in the future.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 26 of its 35 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
13 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  3. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  4. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  5. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  6. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  7. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  8. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  9. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  10. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  11. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  12. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  13. 2023Extending the Frontier of ChatGPT: Code Generation and Debuggingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 8; 2026: 1.

25 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  4. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  5. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  6. 2024Can Language Models Solve Olympiad Programming?cited by 72
  7. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  8. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  9. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  10. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
  11. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  12. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 29
13 more
  1. 2026TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agentscited by 29
  2. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  3. 2024Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformercited by 19
  4. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
  5. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16
  6. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 12
  7. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solvingcited by 6
  8. 2025Exploring Large Language Models for Translating Romanian Computational Problems into Englishcited by 2
  9. 2024CAP: Data Contamination Detection via Consistency Amplificationcited by 1
  10. 2025Computational Reasoning of Large Language Modelscited by 1
  11. 2024Automated Discovery of Integral with Deep Learningcited by 0
  12. 2024"Turing Tests" For An AI Scientistcited by 0
  13. 2024Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
  2. 2025Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
  3. 2026Where Do Large Language Models Fail on Competitive Programming? A Taxonomy of Failures by Algorithm Type and Difficulty Rating ↗
  4. 2024Are Your LLMs Capable of Stable Reasoning?
  5. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programming
  6. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
  7. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
  8. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models