# RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

arXiv: https://arxiv.org/abs/2606.18950 · HTML: https://mlsanity.com/paper/2606.18950/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 12 of its 57 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 12 of 12)

- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Strategic Reasoning with Language Models](https://mlsanity.com/paper/2305.19165/) (2023)
- [AvalonBench: Evaluating LLMs Playing the Game of Avalon](https://mlsanity.com/paper/2310.05036/) (2023)
- [GameGPT: Multi-agent Collaborative Framework for Game Development](https://mlsanity.com/paper/2310.08067/) (2023)
- [PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Models](https://mlsanity.com/paper/2402.01118/) (2024)
- [SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?](https://mlsanity.com/paper/2503.12349/) (2025)
- [Measuring General Intelligence with Generated Games](https://mlsanity.com/paper/2505.07215/) (2025)
- [TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft](https://mlsanity.com/paper/2412.05255/) (2024)
- [AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning](https://mlsanity.com/paper/2511.19304/) (2025)
- [SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models](https://mlsanity.com/paper/2401.17749/) (2024)
- [OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics](https://mlsanity.com/paper/2606.09826/) (2026)
- [SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks](https://mlsanity.com/paper/2508.10428/) (2025)

## Abstract (arXiv)

Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., anticipating and influencing other agents' actions, under uncertainty in competitive and cooperative settings. Real-time strategy (RTS) games can be a natural testbed for diagnosing this limitation, as they demand coordination with allies, adaptation to opponents' strategy, and long-horizon planning under partial observability. However, existing RTS benchmarks offer limited evaluation scope, lack systematic competency diagnosis, and remain fixed in the pre-designed scenario coverage. To address these limitations, we present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield that demands broader strategy diversity than the existing testbeds. The proposed benchmark provides evaluations through diverse gameplay across various matchup structures, diagnostic assessment via mini-games, each targeting an individual strategic competency, and extensible coverage via a self-evolving generation framework that converts free-form queries into new mini-games, improving over successive cycles. Additionally, for VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory. We empirically validate that multiple state-of-the-art VLMs do not perform well when matchups demand tighter coordination, multiagent coordination and when task scale increases.