# Divide-and-Conquer Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.09874 · HTML: https://mlsanity.com/paper/1711.09874/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 30 of 48)

- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)
- [RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning](https://mlsanity.com/paper/2412.09858/) (2024)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity](https://mlsanity.com/paper/2211.01500/) (2022)
- [Specializing Versatile Skill Libraries using Local Mixture of Experts](https://mlsanity.com/paper/2112.04216/) (2021)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners](https://mlsanity.com/paper/2505.23150/) (2025)
- [Improving Policy Optimization with Generalist-Specialist Learning](https://mlsanity.com/paper/2206.12984/) (2022)
- [Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble](https://mlsanity.com/paper/2205.09284/) (2022)
- [PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitation](https://mlsanity.com/paper/2404.01081/) (2024)
- [Hierarchical Neural Dynamic Policies](https://mlsanity.com/paper/2107.05627/) (2021)
- [Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation](https://mlsanity.com/paper/2109.00525/) (2021)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [Multi-Agent Policy Transfer via Task Relationship Modeling](https://mlsanity.com/paper/2203.04482/) (2022)
- [Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach](https://mlsanity.com/paper/2210.12624/) (2022)
- [Multi-objective Reinforcement learning from AI Feedback](https://mlsanity.com/paper/2406.07295/) (2024)
- [Hierarchically Structured Task-Agnostic Continual Learning](https://mlsanity.com/paper/2211.07725/) (2022)
- [Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot](https://mlsanity.com/paper/2307.16503/) (2023)
- [Mixture-of-Variational-Experts for Continual Learning](https://mlsanity.com/paper/2110.12667/) (2021)
- [Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization](https://mlsanity.com/paper/2207.14561/) (2022)
- [Training Transition Policies via Distribution Matching for Complex Tasks](https://mlsanity.com/paper/2110.04357/) (2021)
- [QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing](https://mlsanity.com/paper/2302.00671/) (2023)
- [Task and Domain Adaptive Reinforcement Learning for Robot Control](https://mlsanity.com/paper/2404.18713/) (2024)

## Abstract (arXiv)

Standard model-free deep reinforcement learning (RL) algorithms sample a new initial state for each trial, allowing them to optimize policies that can perform well even in highly stochastic environments. However, problems that exhibit considerable initial state variation typically produce high-variance gradient estimates for model-free RL, making direct policy or value function optimization challenging. In this paper, we develop a novel algorithm that instead partitions the initial state space into "slices", and optimizes an ensemble of policies, each on a different slice. The ensemble is gradually unified into a single policy that can succeed on the whole state space. This approach, which we term divide-and-conquer RL, is able to solve complex tasks where conventional deep RL methods are ineffective. Our results show that divide-and-conquer RL greatly outperforms conventional policy gradient methods on challenging grasping, manipulation, and locomotion tasks, and exceeds the performance of a variety of prior methods. Videos of policies learned by our algorithm can be viewed at http://bit.ly/dnc-rl