# AutoCode: LLMs as Problem Setters for Competitive Programming

arXiv: https://arxiv.org/abs/2510.12803 · HTML: https://mlsanity.com/paper/2510.12803/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 25 of 25)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [TACO: Topics in Algorithmic COde generation dataset](https://mlsanity.com/paper/2312.14852/) (2023)
- [AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2505.16400/) (2025)
- [Dream-Coder 7B: An Open Diffusion Language Model for Code](https://mlsanity.com/paper/2509.01142/) (2025)
- [LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?](https://mlsanity.com/paper/2506.11928/) (2025)
- [rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset](https://mlsanity.com/paper/2505.21297/) (2025)
- [CodeContests+: High-Quality Test Case Generation for Competitive Programming](https://mlsanity.com/paper/2506.05817/) (2025)
- [SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning](https://mlsanity.com/paper/2504.19162/) (2025)
- [ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers](https://mlsanity.com/paper/2305.14591/) (2023)
- [HardTests: Synthesizing High-Quality Test Cases for LLM Coding](https://mlsanity.com/paper/2505.24098/) (2025)
- [ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming](https://mlsanity.com/paper/2505.16667/) (2025)
- [Rethinking Verification for LLM Code Generation: From Generation to Testing](https://mlsanity.com/paper/2507.06920/) (2025)
- [Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems](https://mlsanity.com/paper/2506.06821/) (2025)
- [Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs](https://mlsanity.com/paper/2406.04460/) (2024)
- [LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming](https://mlsanity.com/paper/2505.15039/) (2025)
- [FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming](https://mlsanity.com/paper/2507.13337/) (2025)

## Built on by (most cited first, 11 of 11)

- [FrontierCS: Evolving Challenges for Evolving Intelligence](https://mlsanity.com/paper/2512.15699/) (2025)
- [BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution](https://mlsanity.com/paper/2606.01286/) (2026)
- [EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics](https://mlsanity.com/paper/2609.27308/) (2026)
- [FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale](https://mlsanity.com/paper/2605.14445/) (2026)
- [Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?](https://mlsanity.com/paper/2603.03202/) (2026)
- [Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming](https://mlsanity.com/paper/2606.12864/) (2026)
- [X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests](https://mlsanity.com/paper/2601.06953/) (2026)
- [Scaling the Scaling Logic: Agentic Meta-Synthesis of Logic Reasoning](https://mlsanity.com/paper/2602.13218/) (2026)
- [MathDuels: A Self-Play Benchmark That Grows](https://mlsanity.com/paper/2604.21916/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Combinatorial Synthesis: Scaling Code RLVR via Atomic Decomposition and Recombination](https://mlsanity.com/paper/2605.31058/) (2026)

## Abstract (arXiv)

Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.