# LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?

arXiv: https://arxiv.org/abs/2506.11928 · HTML: https://mlsanity.com/paper/2506.11928/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2025.

## Built on by (most cited first, 30 of 55)

- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models](https://mlsanity.com/paper/2512.13607/) (2025)
- [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation](https://mlsanity.com/paper/2602.16763/) (2026)
- [GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging](https://mlsanity.com/paper/2508.18993/) (2025)
- [RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?](https://mlsanity.com/paper/2509.21016/) (2025)
- [FrontierCS: Evolving Challenges for Evolving Intelligence](https://mlsanity.com/paper/2512.15699/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding](https://mlsanity.com/paper/2512.04111/) (2025)
- [When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions](https://mlsanity.com/paper/2507.20439/) (2025)
- [AutoCode: LLMs as Problem Setters for Competitive Programming](https://mlsanity.com/paper/2510.12803/) (2025)
- [Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases](https://mlsanity.com/paper/2512.10398/) (2025)
- [How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data](https://mlsanity.com/paper/2604.14164/) (2026)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [Scaling Agentic Verifier for Competitive Coding](https://mlsanity.com/paper/2602.04254/) (2026)
- [Nanbeige4.1-3B: A Small General Model that Reasons, Aligns, and Acts](https://mlsanity.com/paper/2602.13367/) (2026)
- [When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets](https://mlsanity.com/paper/2510.00332/) (2025)
- [LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?](https://mlsanity.com/paper/2510.09595/) (2025)
- [Sakana Fugu Technical Report](https://mlsanity.com/paper/2606.21228/) (2026)
- [ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025](https://mlsanity.com/paper/2511.16205/) (2025)
- [Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems](https://mlsanity.com/paper/2602.06319/) (2026)
- [OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation](https://mlsanity.com/paper/2605.15177/) (2026)
- [CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming](https://mlsanity.com/paper/2605.24693/) (2026)
- [PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization](https://mlsanity.com/paper/2607.19653/) (2026)
- [SWE-IF: Aligning Code Evaluation with Human Preference](https://mlsanity.com/paper/2510.07315/) (2025)
- [Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense](https://mlsanity.com/paper/2510.16259/) (2025)
- [K-EXAONE Technical Report](https://mlsanity.com/paper/2601.01739/) (2026)
- [VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?](https://mlsanity.com/paper/2603.15921/) (2026)
- [Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision](https://mlsanity.com/paper/2604.10182/) (2026)
- [LEAF: A Living Benchmark for Event-Augmented Forecasting](https://mlsanity.com/paper/2605.16358/) (2026)

## Abstract (arXiv)

Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning.