# ArtifactArena: Evaluating Models by What They Build in the Physical World

arXiv: https://arxiv.org/abs/2610.06511 · HTML: https://mlsanity.com/paper/2610.06511/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [CodeClash: Benchmarking Goal-Oriented Software Engineering](https://mlsanity.com/paper/2511.00839/) (2025)
- [RoboMoRe: LLM-based Robot Co-design via Joint Optimization of Morphology and Reward](https://mlsanity.com/paper/2506.00276/) (2025)
- [Compositional Machine Design as Program Synthesis with LLMs](https://mlsanity.com/paper/2510.14980/) (2025)
- [SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering](https://mlsanity.com/paper/2502.01860/) (2025)

## Abstract (arXiv)

To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environments. We introduce textscArtifactArena, an open-ended platform where models face a physically grounded hardware-software co-design challenge: engineering fully functional robots to compete in a simulated arena. We evaluate a frontier model's zero-shot, verifier guided refinement, and open-ended physical design capabilities through three harnesses that refine their bot artifacts based on text descriptions, physics simulator feedback, and gameplay data. We benchmark these capabilities with an Elo ranking of frontier models derived from head-to-head tournaments between their artifacts. By releasing this framework and tournament infrastructure for ongoing community submissions, we establish a living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world. Please visit hrefhttps://artifactarena.aihttps://artifactarena.ai for more information.