mlsanityData to 14 Oct 2026

LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?

arXiv 2506.11928 ↗First posted 13 Jun 2025cs.SE cs.AI cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jun 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 35.

55 papers.

Sort:

By area:

  1. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  2. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  3. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  4. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  5. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  6. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 23
  7. 2026Large Language Model Reasoning Failurescited by 21
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  9. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 12
  10. 2025When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptionscited by 11
  11. 2025AutoCode: LLMs as Problem Setters for Competitive Programmingcited by 11
  12. 2025Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebasescited by 11
38 more
  1. 2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 8
  2. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  3. 2026Scaling Agentic Verifier for Competitive Codingcited by 7
  4. 2026Nanbeige4.1-3B: A Small General Model that Reasons, Aligns, and Actscited by 6
  5. 2025When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Marketscited by 5
  6. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 4
  7. 2026Sakana Fugu Technical Reportcited by 4
  8. 2025ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025cited by 3
  9. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  10. 2026OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregationcited by 3
  11. 2026CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programmingcited by 3
  12. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 3
  13. 2025SWE-IF: Aligning Code Evaluation with Human Preferencecited by 2
  14. 2025Distractor Injection Attacks on Large Reasoning Models: Characterization and Defensecited by 2
  15. 2026K-EXAONE Technical Reportcited by 2
  16. 2026VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?cited by 2
  17. 2026Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decisioncited by 2
  18. 2026LEAF: A Living Benchmark for Event-Augmented Forecastingcited by 2
  19. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programmingcited by 2
  20. 2026Multi-LCB: Extending LiveCodeBench to Multiple Programming Languagescited by 2
  21. 2026AI-Assisted Discovery of Convex Relaxations via Dual Agentscited by 2
  22. 2026GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Taskscited by 2
  23. 2025UniCode: Augmenting Evaluation for Code Reasoningcited by 1
  24. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometrycited by 1
  25. 2025HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoningcited by 1
  26. 2026AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?cited by 1
  27. 2026TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesiscited by 1
  28. 2026MathDuels: A Self-Play Benchmark That Growscited by 1
  29. 2026Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMscited by 1
  30. 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1
  31. 2026Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimizationcited by 1
  32. 2026AxDafny: Agentic Verified Code Generation in Dafnycited by 1
  33. 2026Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Sciencecited by 1
  34. 2025Validity Is What You Needcited by 0
  35. 2025LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamicscited by 0
  36. 2026Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMscited by 0
  37. 2026MAS-Algorithm: A Workflow for Solving Algorithmic Programming Problems with a Multi-Agent Systemcited by 0
  38. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
  2. 2024Can Language Models Solve Olympiad Programming?
  3. 2025Evaluating and Improving Large Language Models for Competitive Program Generation
  4. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
  6. 2025OJBench: A Competition Level Code Benchmark For Large Language Models
  7. 2026Where Do Large Language Models Fail on Competitive Programming? A Taxonomy of Failures by Algorithm Type and Difficulty Rating ↗
  8. 2025ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests