# Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators

arXiv: https://arxiv.org/abs/2305.03270 · HTML: https://mlsanity.com/paper/2305.03270/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2023.

## Built on by (most cited first, 21 of 21)

- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [On Bringing Robots Home](https://mlsanity.com/paper/2311.16098/) (2023)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [Emergence of In-Context Reinforcement Learning from Noise Distillation](https://mlsanity.com/paper/2312.12275/) (2023)
- [Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration](https://mlsanity.com/paper/2305.19476/) (2023)
- [Continuously Improving Mobile Manipulation with Autonomous Real-World RL](https://mlsanity.com/paper/2409.20568/) (2024)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies](https://mlsanity.com/paper/2605.00416/) (2026)
- [Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward](https://mlsanity.com/paper/2508.11143/) (2025)
- [Robot Fleet Learning via Policy Merging](https://mlsanity.com/paper/2310.01362/) (2023)
- [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](https://mlsanity.com/paper/2512.13100/) (2025)
- [SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training](https://mlsanity.com/paper/2506.04147/) (2025)
- [Autonomy 2.0: The Quest for Economies of Scale](https://mlsanity.com/paper/2307.03973/) (2023)
- [OpenBot-Fleet: A System for Collective Learning with Real Robots](https://mlsanity.com/paper/2405.07515/) (2024)
- [Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation](https://mlsanity.com/paper/2511.19647/) (2025)
- [DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning](https://mlsanity.com/paper/2606.19656/) (2026)
- [Robustness of Robotic Manipulation: Foundations and Frontiers](https://mlsanity.com/paper/2606.31494/) (2026)

## Abstract (arXiv)

We describe a system for deep reinforcement learning of robotic manipulation skills applied to a large-scale real-world task: sorting recyclables and trash in office buildings. Real-world deployment of deep RL policies requires not only effective training algorithms, but the ability to bootstrap real-world training and enable broad generalization. To this end, our system combines scalable deep RL from real-world data with bootstrapping from training in simulation, and incorporates auxiliary inputs from existing computer vision systems as a way to boost generalization to novel objects, while retaining the benefits of end-to-end training. We analyze the tradeoffs of different design decisions in our system, and present a large-scale empirical validation that includes training on real-world data gathered over the course of 24 months of experimentation, across a fleet of 23 robots in three office buildings, with a total training set of 9527 hours of robotic experience. Our final validation also consists of 4800 evaluation trials across 240 waste station configurations, in order to evaluate in detail the impact of the design decisions in our system, the scaling effects of including more real-world data, and the performance of the method on novel objects. The projects website and videos can be found at hrefhttp://rl-at-scale.github.iorl-at-scale.github.io.