mlsanityData to 14 Oct 2026

Evaluating Large Language Models as Live Strategic Agents: Provider Performance, Hybrid Decomposition, and Operational Gaps in Timed Risk Play

arXiv 2605.22238 ↗First posted 21 May 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Static benchmarks capture only part of how large language models behave in practice. Real systems place models inside repeated loops with time limits, formatting constraints, and failure modes. We study this setting in a timed multi-phase Risk environment with explicit victory targets and repeated planning and execution cycles. In a replicated 32-game cross-provider championship under frozen rules, gemini-3.1-pro-preview won 20 of 32 games against gpt-5.1, claude-opus-4-7, and kimi-k2.6, and the pooled winner distribution differs strongly from an equal-strength null (p approx 1.5 x 10^-5). We then separate planning from execution by standardizing execution on a cheaper Gemini Flash scaffold. Under this design, a pooled 32-game planner bakeoff is consistent with near-equality (p approx 0.821), which indicates that much of the earlier provider spread came from end-to-end system behavior rather than planning alone. To study mechanism, we analyze saved planning and execution traces from the provider championship. Gemini refers to the terminal objective far more often than the other models and increases that focus as victory approaches. Gemini also converts more turns into deep conquest chains, even though it is not the cleanest runtime. These results show that live-agent performance depends on objective tracking, execution conversion, cost, and runtime reliability, and they support evaluating LLMs as components in bounded workflows rather than as isolated benchmark respondents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 6 of its 9 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

6 of 9 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 5

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  4. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 95
  5. 2023Strategic Reasoning with Language Modelscited by 64

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
  2. 2025Playing games with Large language models: Randomness and strategy
  3. 2026GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
  4. 2026Choosing LLMs for Strategic Agent Interactions
  5. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
  6. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  7. 2025Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games
  8. 2024Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games