# Curriculum RL meets Monte Carlo Planning: Optimization of a Real World Container Management Problem

arXiv: https://arxiv.org/abs/2503.17194 · HTML: https://mlsanity.com/paper/2503.17194/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 6 of 6)

- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II](https://mlsanity.com/paper/2012.13169/) (2020)
- [ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource Allocation](https://mlsanity.com/paper/2307.02991/) (2023)
- [Storehouse: a Reinforcement Learning Environment for Optimizing Warehouse Management](https://mlsanity.com/paper/2207.03851/) (2022)
- [Solving a Real-World Optimization Problem Using Proximal Policy Optimization with Curriculum Learning and Reward Engineering](https://mlsanity.com/paper/2404.02577/) (2024)

## Abstract (arXiv)

In this work, we augment reinforcement learning with an inference-time collision model to ensure safe and efficient container management in a waste-sorting facility with limited processing capacity. Each container has two optimal emptying volumes that trade off higher throughput against overflow risk. Conventional reinforcement learning (RL) approaches struggle under delayed rewards, sparse critical events, and high-dimensional uncertainty -- failing to consistently balance higher-volume empties with the risk of safety-limit violations. To address these challenges, we propose a hybrid method comprising: (1) a curriculum-learning pipeline that incrementally trains a PPO agent to handle delayed rewards and class imbalance, and (2) an offline pairwise collision model used at inference time to proactively avert collisions with minimal online cost. Experimental results show that our targeted inference-time collision checks significantly improve collision avoidance, reduce safety-limit violations, maintain high throughput, and scale effectively across varying container-to-PU ratios. These findings offer actionable guidelines for designing safe and efficient container-management systems in real-world facilities.