mlsanityData to 14 Oct 2026

Measuring General Intelligence with Generated Games

arXiv 2505.07215 ↗First posted 12 May 2025cs.AIcited by 24

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We present gg-bench, a collection of game environments designed to evaluate general reasoning capabilities in language models. Unlike most static benchmarks, gg-bench is a data generating process where new evaluation instances can be generated at will. In particular, gg-bench is synthetically generated by (1) using a large language model (LLM) to generate natural language descriptions of novel games, (2) using the LLM to implement each game in code as a Gym environment, and (3) training reinforcement learning (RL) agents via self-play on the generated games. We evaluate language models by their winrate against these RL agents by prompting models with the game description, current board state, and a list of valid moves, after which models output the moves they wish to take. gg-bench is challenging: state-of-the-art LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench using in-context learning, while reasoning models such as o1, o3-mini and DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games, data generation process, and evaluation code in order to support future modeling work and expansion of our benchmark.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 40 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

20 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
  2. 2023Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain21 Oct

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  9. 2022PAL: Program-aided Language Modelscited by 652
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  12. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
6 more
  1. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  2. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  3. 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
  4. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  5. 2024Discovering and exploring cases of educational source code plagiarism with Doloscited by 6
  6. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 14.

24 papers.

Sort:

By area:

  1. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  2. 2025Code World Models for General Game Playingcited by 26
  3. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 22
  4. 2025Assessing Adaptive World Models in Machines with Novel Gamescited by 20
  5. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learningcited by 20
  6. 2026AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Gamescited by 12
  7. 2025Towards Strategic Persuasion with Language Modelscited by 10
  8. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  9. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluationcited by 7
  10. 2025Jailbreak Distillation: Renewable Safety Benchmarkingcited by 5
  11. 2026ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Modelscited by 5
  12. 2026Distilling Game Code World Model Generation into Lightweight Large Language Modelscited by 5
12 more
  1. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  2. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 3
  3. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  4. 2026Scalable Environments Drive Generalizable Agentscited by 2
  5. 2026Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Gamescited by 2
  6. 2026Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planningcited by 1
  7. 2026HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignmentcited by 0
  8. 2026GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Modelscited by 0
  9. 2026RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Modelscited by 0
  10. 2026Witness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environmentscited by 0
  11. 2026Modular Discovery of General Game-Playing Algorithms with Large Language Modelscited by 0
  12. 2026GameSmith: Scaling Zero-Sum Self-Play with Synthesised Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
  2. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  3. 2024GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps
  4. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  5. 2025MastermindEval: A Simple But Scalable Reasoning Benchmark
  6. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors
  7. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
  8. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play