mlsanityData to 7 Oct 2026

ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering

arXiv 2506.09050 ↗First posted 10 Jun 2025cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 18 of its 49 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

18 of 49 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  12. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
6 more
  1. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  2. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  3. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  4. 2024Can Language Models Solve Olympiad Programming?cited by 70
  5. 2024Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspectivecited by 16
  6. 2024Amplifying human performance in combinatorial competitive programmingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 23.

30 papers.

Sort:

By area:

  1. 2026EvoX: Meta-Evolution for Automated Discoverycited by 45
  2. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  3. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  4. 2025The FM Agentcited by 20
  5. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 19
  6. 2026SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfacescited by 15
  7. 2026KAPSO: A Knowledge-grounded framework for Autonomous Program Synthesis and Optimizationcited by 10
  8. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  9. 2026CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economiescited by 7
  10. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 6
  11. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
  12. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
18 more
  1. 2025Evolving Excellence: Automated Optimization of LLM-based Agentscited by 4
  2. 2026Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Taskscited by 4
  3. 2026MirrorCode: AI can rebuild entire programs from behavior alonecited by 4
  4. 2025FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programmingcited by 3
  5. 2026FrontierSmith: Synthesizing Open-Ended Coding Problems at Scalecited by 3
  6. 2026What Do Evolutionary Coding Agents Evolve?cited by 3
  7. 2026Recursive self-improvement of AI research agentscited by 3
  8. 2026SwarmResearch: Orchestrating Coding Agents for Open-Ended Discoverycited by 2
  9. 2026SMCEvolve: Principled Scientific Discovery via Sequential Monte Carlo Evolutioncited by 1
  10. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learningcited by 1
  11. 2026The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testingcited by 1
  12. 2026OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spacescited by 0
  13. 2026Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networkscited by 0
  14. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0
  15. 2026BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problemscited by 0
  16. 2026When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysiscited by 0
  17. 2026LLMs as Adaptive Meta-Solvers: Strategy-Diverse RL for Industrial-Scale Optimizationcited by 0
  18. 2026AgentTime: Can Agents Estimate and Control Their Own Runtime?cited by 0