mlsanityData to 7 Oct 2026

Do You Get the Hint? Benchmarking LLMs on the Board Game Concept

arXiv 2510.13271 ↗First posted 15 Oct 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning. Our results show that this game, easily solved by humans (with a success rate of over 90%), is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). Specifically, we observe that LLMs struggle with interpreting other players' strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 61 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

18 of 61 references.

References · 18

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  3. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  4. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 171
  5. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  6. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 82
  7. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  8. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  9. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  10. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  11. 2025CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generationcited by 9
  12. 2024Are LLMs Good Cryptic Crossword Solvers?cited by 7
6 more
  1. 2021Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Textcited by 6
  2. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agentscited by 5
  3. 2025Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Taskscited by 5
  4. 2025SPaRC: A Spatial Pathfinding Reasoning Challengecited by 5
  5. 2025Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysiscited by 5
  6. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gymcited by 1
  2. 2026Clueing up LLMs with Tool-Augmented Deductive Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  2. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  3. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  4. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  5. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
  6. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules
  7. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  8. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making