# Learning Design and Construction with Varying-Sized Materials via Prioritized Memory Resets

arXiv: https://arxiv.org/abs/2204.05509 · HTML: https://mlsanity.com/paper/2204.05509/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [Self-Supervised Policy Adaptation during Deployment](https://mlsanity.com/paper/2007.04309/) (2020)
- [Phasic Policy Gradient](https://mlsanity.com/paper/2009.04416/) (2020)
- [Near-Optimal Representation Learning for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1810.01257/) (2018)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs](https://mlsanity.com/paper/2012.07277/) (2020)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning](https://mlsanity.com/paper/1912.11032/) (2019)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)

## Built on by (most cited first, 2 of 2)

- [Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning](https://mlsanity.com/paper/2310.04796/) (2023)
- [Learning to Build: Autonomous Robotic Assembly of Stable Structures Without Predefined Plans](https://mlsanity.com/paper/2602.23934/) (2026)

## Abstract (arXiv)

Can a robot autonomously learn to design and construct a bridge from varying-sized blocks without a blueprint? It is a challenging task with long horizon and sparse reward -- the robot has to figure out physically stable design schemes and feasible actions to manipulate and transport blocks. Due to diverse block sizes, the state space and action trajectories are vast to explore. In this paper, we propose a hierarchical approach for this problem. It consists of a reinforcement-learning designer to propose high-level building instructions and a motion-planning-based action generator to manipulate blocks at the low level. For high-level learning, we develop a novel technique, prioritized memory resetting (PMR) to improve exploration. PMR adaptively resets the state to those most critical configurations from a replay buffer so that the robot can resume training on partial architectures instead of from scratch. Furthermore, we augment PMR with auxiliary training objectives and fine-tune the designer with the locomotion generator. Our experiments in simulation and on a real deployed robotic system demonstrate that it is able to effectively construct bridges with blocks of varying sizes at a high success rate. Demos can be found at https://sites.google.com/view/bridge-pmr.