# ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering

arXiv: https://arxiv.org/abs/2506.09050 · HTML: https://mlsanity.com/paper/2506.09050/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective](https://mlsanity.com/paper/2404.06492/) (2024)
- [Amplifying human performance in combinatorial competitive programming](https://mlsanity.com/paper/2411.19744/) (2024)

## Built on by (most cited first, 30 of 39)

- [EvoX: Meta-Evolution for Automated Discovery](https://mlsanity.com/paper/2602.23413/) (2026)
- [SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents](https://mlsanity.com/paper/2605.21384/) (2026)
- [AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?](https://mlsanity.com/paper/2507.15887/) (2025)
- [FrontierCS: Evolving Challenges for Evolving Intelligence](https://mlsanity.com/paper/2512.15699/) (2025)
- [The FM Agent](https://mlsanity.com/paper/2510.26144/) (2025)
- [SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces](https://mlsanity.com/paper/2605.12015/) (2026)
- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [KAPSO: A Knowledge-grounded framework for Autonomous Program Synthesis and Optimization](https://mlsanity.com/paper/2601.21526/) (2026)
- [Recursive self-improvement of AI research agents](https://mlsanity.com/paper/2609.26457/) (2026)
- [TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration](https://mlsanity.com/paper/2604.14116/) (2026)
- [CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies](https://mlsanity.com/paper/2606.16613/) (2026)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks](https://mlsanity.com/paper/2606.29082/) (2026)
- [MirrorCode: AI can rebuild entire programs from behavior alone](https://mlsanity.com/paper/2606.30182/) (2026)
- [SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery](https://mlsanity.com/paper/2607.02807/) (2026)
- [OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling](https://mlsanity.com/paper/2601.19924/) (2026)
- [Evolving Excellence: Automated Optimization of LLM-based Agents](https://mlsanity.com/paper/2512.09108/) (2025)
- [FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming](https://mlsanity.com/paper/2507.13337/) (2025)
- [FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale](https://mlsanity.com/paper/2605.14445/) (2026)
- [What Do Evolutionary Coding Agents Evolve?](https://mlsanity.com/paper/2605.20086/) (2026)
- [OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces](https://mlsanity.com/paper/2605.08904/) (2026)
- [SMCEvolve: Principled Scientific Discovery via Sequential Monte Carlo Evolution](https://mlsanity.com/paper/2605.15308/) (2026)
- [When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis](https://mlsanity.com/paper/2609.15309/) (2026)
- [Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning](https://mlsanity.com/paper/2607.21971/) (2026)
- [The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing](https://mlsanity.com/paper/2608.09855/) (2026)
- [Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networks](https://mlsanity.com/paper/2606.01975/) (2026)
- [Reinforcement Learning without Ground-Truth Solutions can Improve LLMs](https://mlsanity.com/paper/2606.27369/) (2026)
- [BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems](https://mlsanity.com/paper/2608.02612/) (2026)
- [LLMs as Adaptive Meta-Solvers: Strategy-Diverse RL for Industrial-Scale Optimization](https://mlsanity.com/paper/2609.34427/) (2026)

## Abstract (arXiv)

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.