mlsanityData to 14 Oct 2026

KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

arXiv 2505.14552 ↗First posted 20 May 2025cs.CL cs.AI cs.LGcited by 7

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM's general reasoning potential. To address this limitation, we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 26 of its 43 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

26 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Measuring General Intelligence with Generated Games12 May
  2. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?16 Mar
  3. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games9 Dec
  4. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models28 Aug

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2024DeepSeek-V3 Technical Reportcited by 811
  10. 2025Humanity's Last Examcited by 491
  11. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  12. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
10 more
  1. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  2. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  3. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  4. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 70
  5. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  6. 2023AvalonBench: Evaluating LLMs Playing the Game of Avaloncited by 47
  7. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 46
  8. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  9. 2025TextArenacited by 27
  10. 2025CryptoX : Compositional Reasoning Evaluation of Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  3. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 11
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
  6. 2025Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interactioncited by 1
  7. 2026Self-Play Search Distillation for Large Language Model Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  2. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  3. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  4. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  5. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  6. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  7. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  8. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules