mlsanityData to 14 Oct 2026

Baba Is AI: Break the Rules to Beat the Benchmark

arXiv 2407.13729 ↗First posted 18 Jul 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

13 of 17 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  8. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  10. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  11. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  12. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
1 more
  1. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  2. 2026Learning to Continually Learn via Meta-learning Agentic Memory Designscited by 18
  3. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  4. 2025DVM: Towards Controllable LLM Agents in Social Deduction Gamescited by 7
  5. 2026Learning Abstractions for Hierarchical Planning in Program-Synthesis Agentscited by 3
  6. 2025Baba is LLM: Reasoning in a Game with Dynamic Rulescited by 2
  7. 2025AutoLibra: Agent Metric Induction from Open-Ended Human Feedbackcited by 1
  8. 2026Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoningcited by 1
  9. 2026Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Trainingcited by 0
  10. 2026What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
  2. 2025Evaluating Language Models' Evaluations of Games
  3. 2026DiG-bench: Discovery in Games
  4. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  5. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
  6. 2025WebGames: Challenging General-Purpose Web-Browsing AI Agents
  7. 2025Measuring General Intelligence with Generated Games
  8. 2023Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills