# Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Games

arXiv: https://arxiv.org/abs/2505.16401 · HTML: https://mlsanity.com/paper/2505.16401/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 23 of 23)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model](https://mlsanity.com/paper/2503.05132/) (2025)
- [Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning](https://mlsanity.com/paper/2405.10292/) (2024)
- [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning](https://mlsanity.com/paper/2505.02835/) (2025)
- [Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search](https://mlsanity.com/paper/2502.02508/) (2025)
- [CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents](https://mlsanity.com/paper/2401.10568/) (2024)
- [Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty](https://mlsanity.com/paper/2404.18909/) (2024)
- [Large Language Models and Video Games: A Preliminary Scoping Review](https://mlsanity.com/paper/2403.02613/) (2024)
- [Generalization in Text-based Games via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2109.09968/) (2021)
- [Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning](https://mlsanity.com/paper/1904.08129/) (2019)
- [PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning](https://mlsanity.com/paper/2502.10906/) (2025)
- [Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations](https://mlsanity.com/paper/2502.10303/) (2025)
- [Grammar and Gameplay-aligned RL for Game Description Generation with LLMs](https://mlsanity.com/paper/2503.15783/) (2025)

## Built on by (most cited first, 4 of 4)

- [SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2506.24119/) (2025)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environments](https://mlsanity.com/paper/2603.23964/) (2026)
- [Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.07178/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been observed to suddenly exhibit advanced reasoning abilities during reinforcement learning (RL), resembling an ``aha moment'' triggered by simple outcome-based rewards. While RL has proven effective in eliciting such breakthroughs in tasks involving mathematics, coding, and vision, it faces significant challenges in multi-scenario games. The diversity of game rules, interaction modes, and environmental complexities often leads to policies that perform well in one scenario but fail to generalize to others. Simply combining multiple scenarios during training introduces additional challenges, such as training instability and poor performance. To overcome these challenges, we propose Divide-Fuse-Conquer, a framework designed to enhance generalization in multi-scenario RL. This approach starts by heuristically grouping games based on characteristics such as rules and difficulties. Specialized models are then trained for each group to excel at games in the group is what we refer to as the divide step. Next, we fuse model parameters from different groups as a new model, and continue training it for multiple groups, until the scenarios in all groups are conquered. Experiments across 18 TextArena games show that Qwen2.5-32B-Align trained with the Divide-Fuse-Conquer strategy reaches a performance level comparable to Claude3.5, achieving 7 wins and 4 draws. We hope our approach can inspire future research on using reinforcement learning to improve the generalization of LLMs.