# Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

arXiv: https://arxiv.org/abs/1912.11032 · HTML: https://mlsanity.com/paper/1912.11032/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 30 of 55)

- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning](https://mlsanity.com/paper/2106.15087/) (2021)
- [DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training](https://mlsanity.com/paper/2504.09710/) (2025)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Optimistic Active Exploration of Dynamical Systems](https://mlsanity.com/paper/2306.12371/) (2023)
- [Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation](https://mlsanity.com/paper/2206.11403/) (2022)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [Entity-Centric Reinforcement Learning for Object Manipulation from Pixels](https://mlsanity.com/paper/2404.01220/) (2024)
- [Policy Architectures for Compositional Generalization in Control](https://mlsanity.com/paper/2203.05960/) (2022)
- [Efficient and Interpretable Robot Manipulation with Graph Neural Networks](https://mlsanity.com/paper/2102.13177/) (2021)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)
- [Compositional Multi-Object Reinforcement Learning with Linear Relation Networks](https://mlsanity.com/paper/2201.13388/) (2022)
- [Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environments](https://mlsanity.com/paper/2205.06311/) (2022)
- [Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.12030/) (2022)
- [Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems](https://mlsanity.com/paper/2105.12899/) (2021)
- [Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning](https://mlsanity.com/paper/2307.09205/) (2023)
- [PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning](https://mlsanity.com/paper/2502.07600/) (2025)
- [A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation](https://mlsanity.com/paper/2211.14296/) (2022)
- [Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2405.03379/) (2024)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Binding Actions to Objects in World Models](https://mlsanity.com/paper/2204.13022/) (2022)
- [What Planning Problems Can A Relational Neural Network Solve?](https://mlsanity.com/paper/2312.03682/) (2023)
- [Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion](https://mlsanity.com/paper/2602.02722/) (2026)
- [Learning to Design and Construct Bridge without Blueprint](https://mlsanity.com/paper/2108.02439/) (2021)
- [Factored World Models for Zero-Shot Generalization in Robotic Manipulation](https://mlsanity.com/paper/2202.05333/) (2022)

## Abstract (arXiv)

Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring knowledge from simpler to complex tasks. We show that graph-based relational architectures overcome this limitation and enable learning of complex tasks when provided with a simple curriculum of tasks with increasing numbers of objects. We demonstrate the utility of our framework on a simulated block stacking task. Starting from scratch, our agent learns to stack six blocks into a tower. Despite using step-wise sparse rewards, our method is orders of magnitude more data-efficient and outperforms the existing state-of-the-art method that utilizes human demonstrations. Furthermore, the learned policy exhibits zero-shot generalization, successfully stacking blocks into taller towers and previously unseen configurations such as pyramids, without any further training.