# GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents

arXiv: https://arxiv.org/abs/2609.22308 · HTML: https://mlsanity.com/paper/2609.22308/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch](https://mlsanity.com/paper/2505.03733/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping](https://mlsanity.com/paper/2411.03292/) (2024)
- [VideoGameBench: Can Vision-Language Models complete popular video games?](https://mlsanity.com/paper/2505.18134/) (2025)
- [GameDevBench: Evaluating Agentic Capabilities Through Game Development](https://mlsanity.com/paper/2602.11103/) (2026)
- [Image2Struct: Benchmarking Structure Extraction for Vision-Language Models](https://mlsanity.com/paper/2410.22456/) (2024)
- [UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization](https://mlsanity.com/paper/2511.08195/) (2025)
- [General Agent Evaluation](https://mlsanity.com/paper/2602.22953/) (2026)
- [IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?](https://mlsanity.com/paper/2509.24709/) (2025)
- [GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments](https://mlsanity.com/paper/2607.03525/) (2026)
- [GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?](https://mlsanity.com/paper/2608.21833/) (2026)
- [JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines](https://mlsanity.com/paper/2606.19830/) (2026)
- [GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers](https://mlsanity.com/paper/2604.02648/) (2026)
- [FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games](https://mlsanity.com/paper/2509.01052/) (2025)
- [VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents](https://mlsanity.com/paper/2605.26144/) (2026)
- [Aligned Multi-View Scripts for Universal Chart-to-Code Generation](https://mlsanity.com/paper/2604.24559/) (2026)
- [Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages](https://mlsanity.com/paper/2606.20517/) (2026)
- [Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives](https://mlsanity.com/paper/2608.08160/) (2026)
- [AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization](https://mlsanity.com/paper/2603.25118/) (2026)

## Abstract (arXiv)

Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research has extensively evaluated the ability of coding agents to generate programs from textual specifications. However, under black-box conditions where neither source code nor documentation is available, it remains underexplored whether an agent can induce the rules solely through visual observation and active interaction and reproduce the target system as a verifiable executable system. To this end, we present GameReplica, a closed-loop evaluation framework for end-to-end black-box game replication that covers the full perception, exploration, induction, reproduction, and verification pipeline. GameReplica comprises 125 tasks spanning 25 games across 5 core mechanism families, with each game instantiated at five difficulty levels. The tasks require an agent to access the target game only through screenshots and an action interface, induce the key visual elements and gameplay rules from pixel feedback and interaction outcomes, and generate a self-contained, runnable game replica that can be automatically verified by an external program. Experiments show that current coding agents still face substantial challenges in end-to-end black-box replication: the best-performing model (Claude Opus 4.8) achieves an overall score of 71.6%, while the remaining models score only 4.0%--42.9%. Further analysis reveals a consistent pattern across all models: visual-fidelity scores are substantially higher than implementation- and rule-consistency scores, indicating that agents replicate visual appearance more readily than game mechanics. The difficulty levels further amplify the performance gap: from L1 to L5, the overall score of weaker agents drops sharply, whereas that of the best-performing agent declines only slightly.