# Mastering Stacking of Diverse Shapes with Large-Scale Iterative Reinforcement Learning on Real Robots

arXiv: https://arxiv.org/abs/2312.11374 · HTML: https://mlsanity.com/paper/2312.11374/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2023.

## Built on by (most cited first, 13 of 13)

- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation](https://mlsanity.com/paper/2408.11812/) (2024)
- [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025)
- [AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World](https://mlsanity.com/paper/2503.24278/) (2025)
- [JUICER: Data-Efficient Imitation Learning for Robotic Assembly](https://mlsanity.com/paper/2404.03729/) (2024)
- [Autonomous Improvement of Instruction Following Skills via Foundation Models](https://mlsanity.com/paper/2407.20635/) (2024)
- [Offline Actor-Critic Reinforcement Learning Scales to Large Models](https://mlsanity.com/paper/2402.05546/) (2024)
- [Learning from negative feedback, or positive feedback or both](https://mlsanity.com/paper/2410.04166/) (2024)
- [Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers](https://mlsanity.com/paper/2504.04395/) (2025)
- [Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations](https://mlsanity.com/paper/2410.14957/) (2024)
- [Exploiting Policy Idling for Dexterous Manipulation](https://mlsanity.com/paper/2508.15669/) (2025)
- [Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement](https://mlsanity.com/paper/2507.06701/) (2025)
- [Preference-Based Long-Horizon Robotic Stacking with Multimodal Large Language Models](https://mlsanity.com/paper/2509.24163/) (2025)

## Abstract (arXiv)

Reinforcement learning solely from an agent's self-generated data is often believed to be infeasible for learning on real robots, due to the amount of data needed. However, if done right, agents learning from real data can be surprisingly efficient through re-using previously collected sub-optimal data. In this paper we demonstrate how the increased understanding of off-policy learning methods and their embedding in an iterative online/offline scheme (``collect and infer'') can drastically improve data-efficiency by using all the collected experience, which empowers learning from real robot experience only. Moreover, the resulting policy improves significantly over the state of the art on a recently proposed real robot manipulation benchmark. Our approach learns end-to-end, directly from pixels, and does not rely on additional human domain knowledge such as a simulator or demonstrations.