# CodeClash: Benchmarking Goal-Oriented Software Engineering

arXiv: https://arxiv.org/abs/2511.00839 · HTML: https://mlsanity.com/paper/2511.00839/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2025.

## Built on by (most cited first, 10 of 10)

- [Toward Training Superintelligent Software Agents through Self-Play SWE-RL](https://mlsanity.com/paper/2512.18552/) (2025)
- [ContextBench: A Benchmark for Context Retrieval in Coding Agents](https://mlsanity.com/paper/2602.05892/) (2026)
- [SorryDB: Can AI Provers Complete Real-World Lean Theorems?](https://mlsanity.com/paper/2603.02668/) (2026)
- [Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks](https://mlsanity.com/paper/2606.12344/) (2026)
- [CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies](https://mlsanity.com/paper/2606.16613/) (2026)
- [Post-training is (Massive) Supervised Learning](https://mlsanity.com/paper/2606.07527/) (2026)
- [Ask-E: An Environment for Calibrated Question Generation](https://mlsanity.com/paper/2608.06933/) (2026)
- [Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization](https://mlsanity.com/paper/2609.18996/) (2026)
- [ArtifactArena: Evaluating Models by What They Build in the Physical World](https://mlsanity.com/paper/2610.06511/) (2026)
- [Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition](https://mlsanity.com/paper/2610.12341/) (2026)

## Abstract (arXiv)

Current benchmarks for coding evaluate language models (LMs) on concrete, well-specified tasks such as fixing specific bugs or writing targeted tests. However, human programmers do not spend all day incessantly addressing isolated tasks. Instead, real-world software development is grounded in the pursuit of high-level goals, like improving user retention or reducing costs. Evaluating whether LMs can also iteratively develop code to better accomplish open-ended objectives without any explicit guidance remains an open challenge. To address this, we introduce CodeClash, a benchmark where LMs compete in multi-round tournaments to build the best codebase for achieving a competitive objective. Each round proceeds in two phases: agents edit their code, then their codebases compete head-to-head in a code arena that determines winners based on objectives like score maximization, resource acquisition, or survival. Whether it's writing notes, scrutinizing documentation, analyzing competition logs, or creating test suites, models must decide for themselves how to improve their codebases both absolutely and against their opponents. We run 1680 tournaments (25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal that while models exhibit diverse development styles, they share fundamental limitations in strategic reasoning. Models also struggle with long-term codebase maintenance, as repositories become progressively messy and redundant. These limitations are stark: top models lose every round against expert human programmers. We open-source CodeClash to advance the study of autonomous, goal-oriented code development.