mlsanityData to 7 Oct 2026

Can Language Models Solve Olympiad Programming?

arXiv 2404.10952 ↗First posted 16 Apr 2024cs.CL cs.AI cs.PLcited by 70

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computing olympiads contain some of the most challenging problems for humans, requiring complex algorithmic reasoning, puzzle solving, in addition to generating efficient code. However, it has been understudied as a domain to evaluate language models (LMs). In this paper, we introduce the USACO benchmark with 307 problems from the USA Computing Olympiad, along with high-quality unit tests, reference code, and official analyses for each problem. These resources enable us to construct and test a range of LM inference methods for competitive programming for the first time. We find GPT-4 only achieves a 8.7% pass@1 accuracy with zero-shot chain-of-thought prompting, and our best inference method improves it to 20.2% using a combination of self-reflection and retrieval over episodic knowledge. However, this is far from solving the benchmark. To better understand the remaining challenges, we design a novel human-in-the-loop study and surprisingly find that a small number of targeted hints enable GPT-4 to solve 13 out of 15 problems previously unsolvable by any model and method. Our benchmark, baseline methods, quantitative results, and qualitative analysis serve as an initial step toward LMs with grounded, creative, and algorithmic reasoning.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

24 of 28 references.

References · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
12 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2021Measuring Coding Challenge Competence With APPScited by 879
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  4. 2023Cognitive Architectures for Language Agentscited by 422
  5. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  6. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  7. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  8. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  9. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  10. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  11. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  12. 2023Explaining Competitive-Level Programming Solutions using LLMscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 39; 2026: 22.

70 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  4. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  5. 2024AI Agents That Mattercited by 199
  6. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  7. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 109
  8. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  9. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  10. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  11. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  12. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
38 more
  1. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  2. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programmingcited by 39
  3. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  4. 2025ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineeringcited by 30
  5. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  6. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  7. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22
  8. 2025OJBench: A Competition Level Code Benchmark For Large Language Modelscited by 22
  9. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  10. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  11. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  12. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  13. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16
  14. 2025ProBench: Benchmarking Large Language Models in Competitive Programmingcited by 15
  15. 2025ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contestscited by 12
  16. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  17. 2025IMPersona: Evaluating Individual Level LM Impersonationcited by 11
  18. 2026Detecting Safety Violations Across Many Agent Tracescited by 11
  19. 2025AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitionscited by 10
  20. 2025Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinkingcited by 10
  21. 2025Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problemscited by 9
  22. 2025Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learningcited by 9
  23. 2026Efficient Benchmarking of AI Agentscited by 9
  24. 2025Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weightscited by 8
  25. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  26. 2025SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoningcited by 6
  27. 2025Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigmcited by 5
  28. 2026Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluationcited by 5
  29. 2026Scaling Agentic Verifier for Competitive Codingcited by 5
  30. 2025When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaborationcited by 4
  31. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
  32. 2025QueST: Incentivizing LLMs to Generate Difficult Problemscited by 4
  33. 2026Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steeringcited by 4
  34. 2025CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programmingcited by 3
  35. 2025Scaling Laws for Code: A More Data-Hungry Regimecited by 3
  36. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 3
  37. 2026Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolutioncited by 3
  38. 2025AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programmingcited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
  2. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
  3. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
  4. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  5. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Models
  6. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
  8. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning