mlsanityData to 7 Oct 2026

When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

arXiv 2607.19523 ↗First posted 21 Jul 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks, but its effect on behavioral diversity in sequential decision-making remains under-explored. We study this question in a controlled suite of deterministic board games based on tic-tac-toe variants, where optimal actions are exactly computable and diversity can be measured directly. Across state-level evaluation, arena gameplay, and training trajectories, we find that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy. Furthermore, standard SFT improves accuracy but often induces premature diversity collapse, which exceeds what is minimally required by the accuracy-diversity tradeoff. We then show that action augmentation, which trains on all optimal actions per state rather than a single demonstrated action, would partially mitigates this effect. Our results identify narrow-support imitation as a source of policy collapse in LLM decision-making and suggest that preserving action support during SFT is important for maintaining exploratory behavior.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 28 of its 36 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

28 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess1 Dec
  2. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play5 Aug
  3. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules23 Jun
  4. 2025TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games11 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  8. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  10. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  11. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  12. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
12 more
  1. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  2. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  3. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  4. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  5. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
  6. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  7. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
  8. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  9. 2025ChessQA: Evaluating Large Language Models for Chess Understandingcited by 8
  10. 2026SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuningcited by 5
  11. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planningcited by 3
  12. 2025Offline Learning of Controllable Diverse Behaviorscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
  2. 2026How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess
  3. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use
  4. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  5. 2026Towards Improving Sequential Decision-Making in LLM Agents via Experience Memory
  6. 2026Can RL Improve Generalization of LLM Agents? An Empirical Study
  7. 2026Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training
  8. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games