# AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games

arXiv: https://arxiv.org/abs/2602.17594 · HTML: https://mlsanity.com/paper/2602.17594/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [Understanding the Weakness of Large Language Model Agents within a Complex Android Environment](https://mlsanity.com/paper/2402.06596/) (2024)
- [General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithms](https://mlsanity.com/paper/1802.10363/) (2018)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [VideoGameBench: Can Vision-Language Models complete popular video games?](https://mlsanity.com/paper/2505.18134/) (2025)
- [General Scales Unlock AI Evaluation with Explanatory and Predictive Power](https://mlsanity.com/paper/2503.06378/) (2025)
- [A Definition of AGI](https://mlsanity.com/paper/2510.18212/) (2025)
- [NitroGen: An Open Foundation Model for Generalist Gaming Agents](https://mlsanity.com/paper/2601.02427/) (2026)
- [Code World Models for General Game Playing](https://mlsanity.com/paper/2510.04542/) (2025)
- [GAVEL: Generating Games Via Evolution and Language Models](https://mlsanity.com/paper/2407.09388/) (2024)
- [Measuring General Intelligence with Generated Games](https://mlsanity.com/paper/2505.07215/) (2025)
- [Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents](https://mlsanity.com/paper/2510.23691/) (2025)
- [Assessing Adaptive World Models in Machines with Novel Games](https://mlsanity.com/paper/2507.12821/) (2025)
- [Word2World: Generating Stories and Worlds through Large Language Models](https://mlsanity.com/paper/2405.06686/) (2024)
- [Benchmarking World-Model Learning with Environment-Level Queries](https://mlsanity.com/paper/2510.19788/) (2025)
- [GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games](https://mlsanity.com/paper/2508.08501/) (2025)
- [Generation and Evaluation in the Human Invention Process through the Lens of Game Design](https://mlsanity.com/paper/2508.10914/) (2025)

## Built on by (most cited first, 12 of 12)

- [GameVerse: Can Vision-Language Models Learn from Video-based Reflection?](https://mlsanity.com/paper/2603.06656/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning](https://mlsanity.com/paper/2608.03571/) (2026)
- [DiG-bench: Discovery in Games](https://mlsanity.com/paper/2608.12593/) (2026)
- [Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games](https://mlsanity.com/paper/2606.17657/) (2026)
- [VISTA: A Visual Harness for Reasoning in an Interactive World](https://mlsanity.com/paper/2610.02200/) (2026)
- [Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks](https://mlsanity.com/paper/2608.03340/) (2026)
- [PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games](https://mlsanity.com/paper/2609.09059/) (2026)
- [Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization](https://mlsanity.com/paper/2609.18996/) (2026)
- [PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence](https://mlsanity.com/paper/2610.07127/) (2026)

## Abstract (arXiv)

Rigorously evaluating machine intelligence against the broad spectrum of human general intelligence has become increasingly important and challenging in this era of rapid technological advance. Conventional AI benchmarks typically assess only narrow capabilities in a limited range of human activity. Most are also static, quickly saturating as developers explicitly or implicitly optimize for them. We propose that a more promising way to evaluate human-like general intelligence in AI systems is through a particularly strong form of general game playing: studying how and how well they play and learn to play all conceivable human games, in comparison to human players with the same level of experience, time, or other resources. We define a "human game" to be a game designed by humans for humans, and argue for the evaluative suitability of this space of all such games people can imagine and enjoy -- the "Multiverse of Human Games". Taking a first step towards this vision, we introduce the AI GameStore, a scalable and open-ended platform that uses LLMs with humans-in-the-loop to synthesize new representative human games, by automatically sourcing and adapting standardized and containerized variants of game environments from popular human digital gaming platforms. As a proof of concept, we generated 100 such games based on the top charts of Apple App Store and Steam, and evaluated seven frontier vision-language models (VLMs) on short episodes of play. The best models achieved less than 10% of the human average score on the majority of the games, and especially struggled with games that challenge world-model learning, memory and planning. We conclude with a set of next steps for building out the AI GameStore as a practical way to measure and drive progress toward human-like general intelligence in machines.