# BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors

arXiv: https://arxiv.org/abs/2602.13214 · HTML: https://mlsanity.com/paper/2602.13214/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents](https://mlsanity.com/paper/2406.06613/) (2024)
- [LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments](https://mlsanity.com/paper/2402.16499/) (2024)
- [R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation](https://mlsanity.com/paper/2505.02018/) (2025)
- [LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models](https://mlsanity.com/paper/2408.15778/) (2024)
- [GAMEBoT: Transparent Assessment of LLM Reasoning in Games](https://mlsanity.com/paper/2412.13602/) (2024)
- [ChessQA: Evaluating Large Language Models for Chess Understanding](https://mlsanity.com/paper/2510.23948/) (2025)
- [A critical review of methods and challenges in large language models](https://mlsanity.com/paper/2404.11973/) (2024)
- [Mxplainer: Explain and Learn Insights by Imitating Mahjong Agents](https://mlsanity.com/paper/2506.14246/) (2025)

## Built on by (most cited first, 2 of 2)

- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaboration](https://mlsanity.com/paper/2603.01260/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in interactive environments requiring strategic decision-making, yet systematic evaluation of these capabilities remains challenging. Existing benchmarks for LLMs primarily assess static reasoning through isolated tasks and fail to capture dynamic strategic abilities. Recent game-based evaluations employ LLM-vs-LLM tournaments that produce relative rankings dependent on transient model pools, incurring quadratic computational costs and lacking stable performance anchors for longitudinal tracking. The central challenge is establishing a scalable evaluation framework that measures LLM strategic reasoning against consistent, interpretable standards rather than volatile peer models. Here we show that anchoring LLM evaluation to fixed hierarchies of skill-calibrated game Artificial Intelligence (AI) enables linear-time absolute skill measurement with stable cross-temporal interpretability. Built on the Botzone platform's established competitive infrastructure, our BotzoneBench evaluates LLMs across eight diverse games spanning deterministic perfect-information board games to stochastic imperfect-information card games. Through systematic assessment of 177,047 state-action pairs from five flagship models, we reveal significant performance disparities and identify distinct strategic behaviors, with top-performing models achieving proficiency comparable to mid-to-high-tier specialized game AI in multiple domains. This anchored evaluation paradigm generalizes beyond games to any domain with well-defined skill hierarchies, establishing a scalable and reusable framework for assessing interactive AI capabilities.