# AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions

arXiv: https://arxiv.org/abs/2508.16402 · HTML: https://mlsanity.com/paper/2508.16402/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 11 of 11)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [CodeContests+: High-Quality Test Case Generation for Competitive Programming](https://mlsanity.com/paper/2506.05817/) (2025)

## Built on by (most cited first, 11 of 11)

- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [TeachLM: Post-Training LLMs for Education Using Authentic Learning Data](https://mlsanity.com/paper/2510.05087/) (2025)
- [VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code](https://mlsanity.com/paper/2510.06296/) (2025)
- [LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?](https://mlsanity.com/paper/2510.09595/) (2025)
- [GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning](https://mlsanity.com/paper/2604.02721/) (2026)
- [Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution](https://mlsanity.com/paper/2605.15301/) (2026)
- [Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming](https://mlsanity.com/paper/2606.12864/) (2026)
- [Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming](https://mlsanity.com/paper/2604.01302/) (2026)
- [Mi:dm K 2.5 Pro](https://mlsanity.com/paper/2603.18788/) (2026)
- [InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information](https://mlsanity.com/paper/2608.29632/) (2026)

## Abstract (arXiv)

Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.