# SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?

arXiv: https://arxiv.org/abs/2503.12349 · HTML: https://mlsanity.com/paper/2503.12349/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 37 of its 102 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 37)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents](https://mlsanity.com/paper/2406.06613/) (2024)
- [LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench](https://mlsanity.com/paper/2409.13373/) (2024)
- [MindAgent: Emergent Gaming Interaction](https://mlsanity.com/paper/2309.09971/) (2023)
- [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023)
- [Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena](https://mlsanity.com/paper/2310.05746/) (2023)
- [LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models](https://mlsanity.com/paper/2310.03903/) (2023)
- [How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments](https://mlsanity.com/paper/2403.11807/) (2024)
- [Game-theoretic LLM: Agent Workflow for Negotiation Games](https://mlsanity.com/paper/2411.05990/) (2024)

## Built on by (most cited first, 24 of 24)

- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [TextArena](https://mlsanity.com/paper/2504.11442/) (2025)
- [MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation](https://mlsanity.com/paper/2505.17123/) (2025)
- [Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People](https://mlsanity.com/paper/2510.20886/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation](https://mlsanity.com/paper/2505.14552/) (2025)
- [One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence](https://mlsanity.com/paper/2602.03109/) (2026)
- [TextAtari: 100K Frames Game Playing with Language Agents](https://mlsanity.com/paper/2506.04098/) (2025)
- [PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting](https://mlsanity.com/paper/2606.08878/) (2026)
- [When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets](https://mlsanity.com/paper/2510.00332/) (2025)
- [MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games](https://mlsanity.com/paper/2603.09022/) (2026)
- [MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs](https://mlsanity.com/paper/2605.29512/) (2026)
- [Language Model as Planner and Formalizer under Constraints](https://mlsanity.com/paper/2510.05486/) (2025)
- [CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observations](https://mlsanity.com/paper/2512.23328/) (2025)
- [SOTOPIA-TOM: Evaluating Privacy and Information Management in Multi-Agent Interaction with Theory of Mind](https://mlsanity.com/paper/2605.02307/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V](https://mlsanity.com/paper/2604.07733/) (2026)
- [TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis](https://mlsanity.com/paper/2602.02523/) (2026)
- [Skill Issue: Are Skills Language-Invariant in LLMs?](https://mlsanity.com/paper/2608.25832/) (2026)
- [When AI Navigates the Fog of War](https://mlsanity.com/paper/2603.16642/) (2026)
- [TUX: Measuring Human--AI Tacit Understanding](https://mlsanity.com/paper/2605.30930/) (2026)
- [MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution](https://mlsanity.com/paper/2606.00017/) (2026)
- [RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models](https://mlsanity.com/paper/2606.18950/) (2026)
- [Position: AI Is Not Ready for Strategic Conflicts](https://mlsanity.com/paper/2609.16189/) (2026)

## Abstract (arXiv)

Reasoning and strategic behavior in social interactions is a hallmark of intelligence. This form of reasoning is significantly more sophisticated than isolated planning or reasoning tasks in static settings (e.g., math problem solving). In this paper, we present Strategic Planning, Interaction, and Negotiation (SPIN-Bench), a new multi-domain evaluation designed to measure the intelligence of strategic planning and social reasoning. While many existing benchmarks focus on narrow planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks, competitive board games, cooperative card games, and multi-agent negotiation scenarios in one unified framework. The framework includes both a benchmark as well as an arena to simulate and evaluate the variety of social settings to test reasoning and strategic behavior of AI agents. We formulate the benchmark SPIN-Bench by systematically varying action spaces, state complexity, and the number of interacting agents to simulate a variety of social settings where success depends on not only methodical and step-wise decision making, but also conceptual inference of other (adversarial or cooperative) participants. Our experiments reveal that while contemporary LLMs handle basic fact retrieval and short-range planning reasonably well, they encounter significant performance bottlenecks in tasks requiring deep multi-hop reasoning over large state spaces and socially adept coordination under uncertainty. We envision SPIN-Bench as a catalyst for future research on robust multi-agent planning, social reasoning, and human--AI teaming. Project Website: https://spinbench.github.io/