# Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers

arXiv: https://arxiv.org/abs/2212.11498 · HTML: https://mlsanity.com/paper/2212.11498/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1906.04737/) (2019)
- [PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning](https://mlsanity.com/paper/1809.03531/) (2018)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [Graph Neural Networks for Decentralized Multi-Robot Path Planning](https://mlsanity.com/paper/1912.06095/) (2019)
- [Lifelong Multi-Agent Path Finding in Large-Scale Warehouses](https://mlsanity.com/paper/2005.07371/) (2020)
- [Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Tasks](https://mlsanity.com/paper/1705.10868/) (2017)
- [PRIMAL2: Pathfinding via Reinforcement and Imitation Multi-Agent Learning -- Lifelong](https://mlsanity.com/paper/2010.08184/) (2020)
- [Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning](https://mlsanity.com/paper/1901.08492/) (2019)
- [Lifelong Path Planning with Kinematic Constraints for Multi-Agent Pickup and Delivery](https://mlsanity.com/paper/1812.06355/) (2018)
- [Macro-Action-Based Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2004.08646/) (2020)
- [Multi-Agent Path Finding with Prioritized Communication Learning](https://mlsanity.com/paper/2202.03634/) (2022)
- [Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems](https://mlsanity.com/paper/2105.12899/) (2021)
- [Multi-Goal Multi-Agent Pickup and Delivery](https://mlsanity.com/paper/2208.01223/) (2022)
- [Cooperative Multi-Agent Path Finding: Beyond Path Planning and Collision Avoidance](https://mlsanity.com/paper/2105.10993/) (2021)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)

## Built on by (most cited first, 30 of 30)

- [EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration](https://mlsanity.com/paper/2601.16489/) (2026)
- [CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale](https://mlsanity.com/paper/2507.05178/) (2025)
- [SMAClite: A Lightweight Environment for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.05566/) (2023)
- [Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03439/) (2023)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Go-SLAM: Grounded Object Segmentation and Localization with Gaussian Splatting SLAM](https://mlsanity.com/paper/2409.16944/) (2024)
- [Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.23604/) (2025)
- [High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning](https://mlsanity.com/paper/2510.20218/) (2025)
- [GTDE: Grouped Training with Decentralized Execution for Multi-agent Actor-Critic](https://mlsanity.com/paper/2501.10367/) (2024)
- [Improving Sample Efficiency in Multi-Agent Reinforcement Learning for Simulated Football Games via Exploration](https://mlsanity.com/paper/2503.13077/) (2025)
- [JaxRobotarium: Training and Deploying Multi-Robot Policies in 10 Minutes](https://mlsanity.com/paper/2505.06771/) (2025)
- [Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.04864/) (2025)
- [Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL](https://mlsanity.com/paper/2505.22151/) (2025)
- [Topology-Aware and Highly Generalizable Deep Reinforcement Learning for Efficient Retrieval in Multi-Deep Storage Systems](https://mlsanity.com/paper/2506.14787/) (2025)
- [Learning Bilateral Team Formation in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.20039/) (2025)
- [Explaining Decentralized Multi-Agent Reinforcement Learning Policies](https://mlsanity.com/paper/2511.10409/) (2025)
- [Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.05407/) (2026)
- [SCSimulator: An Exploratory Visual Analytics Framework for Partner Selection in Supply Chains through LLM-driven Multi-Agent Simulation](https://mlsanity.com/paper/2601.14566/) (2026)
- [NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.06977/) (2026)
- [SOAR: Real-Time Joint Optimization of Order Allocation and Robot Scheduling in Robotic Mobile Fulfillment Systems](https://mlsanity.com/paper/2605.03842/) (2026)
- [Coordination Matters: Evaluation of Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.06557/) (2026)
- [Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems](https://mlsanity.com/paper/2605.08758/) (2026)
- [LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.29293/) (2026)
- [Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.09610/) (2026)
- [Offline Reinforcement Learning for Warehouse SLAM Throughput Control](https://mlsanity.com/paper/2606.23978/) (2026)
- [A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturing](https://mlsanity.com/paper/2608.07148/) (2026)
- [Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.05298/) (2026)
- [Integrated Optimization of Automated Warehouse Operations and Last-Mile Transport for Differentiated On-Demand Delivery](https://mlsanity.com/paper/2609.19048/) (2026)
- [CoLA: Multi-VLA Coordination via Learned Adapters](https://mlsanity.com/paper/2610.15337/) (2026)
- [Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL](https://mlsanity.com/paper/2610.17409/) (2026)

## Abstract (arXiv)

We consider a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. The fundamental problem we tackle, called the order-picking problem, is how these worker agents must coordinate their movement and actions in the warehouse to maximise performance in this task. Established industry methods using heuristic approaches require large engineering efforts to optimise for innately variable warehouse configurations. In contrast, multi-agent reinforcement learning (MARL) can be flexibly applied to diverse warehouse configurations (e.g. size, layout, number/types of workers, item replenishment frequency), and different types of order-picking paradigms (e.g. Goods-to-Person and Person-to-Goods), as the agents can learn how to cooperate optimally through experience. We develop hierarchical MARL algorithms in which a manager agent assigns goals to worker agents, and the policies of the manager and workers are co-trained toward maximising a global objective (e.g. pick rate). Our hierarchical algorithms achieve significant gains in sample efficiency over baseline MARL algorithms and overall pick rates over multiple established industry heuristics in a diverse set of warehouse configurations and different order-picking paradigms.