# Scalable Hierarchical Reinforcement Learning for Hyper Scale Multi-Robot Task Planning

arXiv: https://arxiv.org/abs/2412.19538 · HTML: https://mlsanity.com/paper/2412.19538/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 55 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Reinforcement Learning for Combinatorial Optimization: A Survey](https://mlsanity.com/paper/2003.03600/) (2020)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning](https://mlsanity.com/paper/2010.12367/) (2020)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Temporal Self-Attention Network for Medical Concept Embedding](https://mlsanity.com/paper/1909.06886/) (2019)
- [Robust Task Scheduling for Heterogeneous Robot Teams under Capability Uncertainty](https://mlsanity.com/paper/2106.12111/) (2021)
- [Speeding up Routing Schedules on Aisle-Graphs with Single Access](https://mlsanity.com/paper/2102.05733/) (2021)
- [Multi-Robot Pickup and Delivery via Distributed Resource Allocation](https://mlsanity.com/paper/2104.02415/) (2021)
- [Adaptive Task Planning for Large-Scale Robotized Warehouses](https://mlsanity.com/paper/2205.00831/) (2022)

## Abstract (arXiv)

To improve the efficiency of warehousing system and meet huge customer orders, we aim to solve the challenges of dimension disaster and dynamic properties in hyper scale multi-robot task planning (MRTP) for robotic mobile fulfillment system (RMFS). Existing research indicates that hierarchical reinforcement learning (HRL) is an effective method to reduce these challenges. Based on that, we construct an efficient multi-stage HRL-based multi-robot task planner for hyper scale MRTP in RMFS, and the planning process is represented with a special temporal graph topology. To ensure optimality, the planner is designed with a centralized architecture, but it also brings the challenges of scaling up and generalization that require policies to maintain performance for various unlearned scales and maps. To tackle these difficulties, we first construct a hierarchical temporal attention network (HTAN) to ensure basic ability of handling inputs with unfixed lengths, and then design multi-stage curricula for hierarchical policy learning to further improve the scaling up and generalization ability while avoiding catastrophic forgetting. Additionally, we notice that policies with hierarchical structure suffer from unfair credit assignment that is similar to that in multi-agent reinforcement learning, inspired of which, we propose a hierarchical reinforcement learning algorithm with counterfactual rollout baseline to improve learning performance. Experimental results demonstrate that our planner outperform other state-of-the-art methods on various MRTP instances in both simulated and real-world RMFS. Also, our planner can successfully scale up to hyper scale MRTP instances in RMFS with up to 200 robots and 1000 retrieval racks on unlearned maps while keeping superior performance over other methods.