# Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition

arXiv: https://arxiv.org/abs/2610.12341 · HTML: https://mlsanity.com/paper/2610.12341/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 23 of 23)

- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation](https://mlsanity.com/paper/2310.02304/) (2023)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents](https://mlsanity.com/paper/2406.06613/) (2024)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [GameArena: Evaluating LLM Reasoning through Live Computer Games](https://mlsanity.com/paper/2412.06394/) (2024)
- [TextArena](https://mlsanity.com/paper/2504.11442/) (2025)
- [AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement](https://mlsanity.com/paper/2608.20318/) (2026)
- [CodeClash: Benchmarking Goal-Oriented Software Engineering](https://mlsanity.com/paper/2511.00839/) (2025)
- [Learning Game-Playing Agents with Generative Code Optimization](https://mlsanity.com/paper/2508.19506/) (2025)
- [From Code to Play: Benchmarking Program Search for Games Using Large Language Models](https://mlsanity.com/paper/2412.04057/) (2024)
- [ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition](https://mlsanity.com/paper/2504.12562/) (2025)
- [CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments](https://mlsanity.com/paper/2510.26852/) (2025)
- [ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas](https://mlsanity.com/paper/2602.04296/) (2026)
- [Scalable Behaviour Cloning on Browser Using via Skill Distillation](https://mlsanity.com/paper/2606.32014/) (2026)
- [EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments](https://mlsanity.com/paper/2607.02440/) (2026)
- [Game Arena: Strategic LLM Evaluation in Competitive Environments](https://mlsanity.com/paper/2609.31473/) (2026)
- [Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning](https://mlsanity.com/paper/2511.20613/) (2025)
- [Heuristic Learning for Active Flow Control Using Coding Agents](https://mlsanity.com/paper/2607.11565/) (2026)

## Abstract (arXiv)

Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain challenging. AI agents offer an alternative by turning game experience into revisions of executable policies. Building on heuristic learning (HL), we formalize Adversarial Heuristic Learning (AHL), a paradigm that uses AI agents as learning engines to refine game policies and supporting software while keeping model weights fixed. We introduce AAArena, a benchmark comprising 12 authentic adversarial games and 1,920 archived human programs, with an evaluation protocol modeled on real-world game competitions. Agents interpret rules, choose opponents, analyze replays, and revise game agents to achieve their highest ranking within fixed match and evaluation budgets. We evaluate valcompletedmodels model and harness configurations: Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders. Performance is generally weaker in games with more complex rule specifications. Further experiments show that opponent selection and dense feedback support policy improvement, and that agents learn from both on-policy replays of their own matches and off-policy replays of other players' matches. These results highlight HL's potential in adversarial games and identify persistent challenges in game understanding, strategy implementation, and long-horizon policy development.