mlsanityData to 7 Oct 2026

GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps

arXiv 2410.07765 ↗First posted 10 Oct 2024cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently demonstrated great success in generating and understanding natural language. While they have also shown potential beyond the domain of natural language, it remains an open question as to what extent and in which way these LLMs can plan. We investigate their planning capabilities by proposing GameTraversalBenchmark (GTB), a benchmark consisting of diverse 2D grid-based game maps. An LLM succeeds if it can traverse through given objectives, with a minimum number of steps and a minimum number of generation errors. We evaluate a number of LLMs on GTB and found that GPT-4-Turbo achieved the highest score of 44.97% on GTBScore (GTBS), a composite score that combines the three above criteria. Furthermore, we preliminarily test large reasoning models, namely o1, which scores 67.84% on GTBS, indicating that the benchmark remains challenging for current models. Code, data, and documentation are available at https://github.com/umair-nasir14/Game-Traversal-Benchmark.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

32 of 39 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  12. 2022A Generalist Agentcited by 802
20 more
  1. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  3. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  4. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  5. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  6. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  7. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  8. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  9. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  10. 2023Level Generation Through Large Language Modelscited by 68
  11. 2019An investigation of model-free planningcited by 64
  12. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
  13. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  14. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  15. 2020The Chess Transformer: Mastering Play using Generative Language Modelscited by 36
  16. 2023Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planningcited by 24
  17. 2023Practical PCG Through Large Language Modelscited by 21
  18. 2024Word2World: Generating Stories and Worlds through Large Language Modelscited by 18
  19. 2020The Go Transformer: Natural Language Modeling for Game Playcited by 12
  20. 2024Will GPT-4 Run DOOM?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  3. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  4. 2025Agents of Change: Self-Evolving LLM Agents for Strategic Planningcited by 20
  5. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 8
  6. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  7. 2025PuzzleJAX: A Benchmark for Reasoning and Learningcited by 4
  8. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  2. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  3. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  4. 2025Code World Models for General Game Playing
  5. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
  6. 2024Mastering Board Games by External and Internal Planning with Language Models
  7. 2025Measuring General Intelligence with Generated Games
  8. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generation