# Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?

arXiv: https://arxiv.org/abs/2506.12713 · HTML: https://mlsanity.com/paper/2506.12713/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 15 Jun 2025.

## Builds on (most cited first, 11 of 11)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://mlsanity.com/paper/2505.18761/) (2025)
- [CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality](https://mlsanity.com/paper/2502.08923/) (2025)
- [Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM](https://mlsanity.com/paper/2505.10861/) (2025)

## Built on by (most cited first, 6 of 6)

- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?](https://mlsanity.com/paper/2510.09595/) (2025)
- [GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning](https://mlsanity.com/paper/2604.02721/) (2026)
- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)
- [When Independent Sampling Outperforms Agentic Reasoning](https://mlsanity.com/paper/2605.08478/) (2026)
- [InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information](https://mlsanity.com/paper/2608.29632/) (2026)

## Abstract (arXiv)

Code generation is a core capability of large language models (LLMs), yet mainstream benchmarks (e.g., APPs and LiveCodeBench) contain questions with medium-level difficulty and pose no challenge to advanced LLMs. To better reflected the advanced reasoning and code generation ability, We introduce Humanity's Last Code Exam (HLCE), comprising 235 most challenging problems from the International Collegiate Programming Contest (ICPC World Finals) and the International Olympiad in Informatics (IOI) spanning 2010 - 2024. As part of HLCE, we design a harmonized online-offline sandbox that guarantees fully reproducible evaluation. Through our comprehensive evaluation, we observe that even the strongest reasoning LLMs: o4-mini(high) and Gemini-2.5 Pro, achieve pass@1 rates of only 15.9% and 11.4%, respectively. Meanwhile, we propose a novel "self-recognition" task to measure LLMs' awareness of their own capabilities. Results indicate that LLMs' self-recognition abilities are not proportionally correlated with their code generation performance. Finally, our empirical validation of test-time scaling laws reveals that current advanced LLMs have substantial room for improvement on complex programming tasks. We expect HLCE to become a milestone challenge for code generation and to catalyze advances in high-performance reasoning and human-AI collaborative programming. Our code and dataset are also public available(https://github.com/Humanity-s-Last-Code-Exam/HLCE).