# Robotic Test Tube Rearrangement Using Combined Reinforcement Learning and Motion Planning

arXiv: https://arxiv.org/abs/2401.09772 · HTML: https://mlsanity.com/paper/2401.09772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning](https://mlsanity.com/paper/1510.08906/) (2015)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [NeRP: Neural Rearrangement Planning for Unknown Objects](https://mlsanity.com/paper/2106.01352/) (2021)
- [Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning](https://mlsanity.com/paper/1904.10348/) (2019)
- [Predicting Stable Configurations for Semantic Placement of Novel Objects](https://mlsanity.com/paper/2108.12062/) (2021)
- [Learning Agent-Aware Affordances for Closed-Loop Interaction with Articulated Objects](https://mlsanity.com/paper/2209.05802/) (2022)
- [Multi-Object Rearrangement with Monte Carlo Tree Search:A Case Study on Planar Nonprehensile Sorting](https://mlsanity.com/paper/1912.07024/) (2019)
- [Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers](https://mlsanity.com/paper/2209.11943/) (2022)
- [Complexity Results and Fast Methods for Optimal Tabletop Rearrangement with Overhand Grasps](https://mlsanity.com/paper/1711.07369/) (2017)
- [Manipulation Planning Among Movable Obstacles Using Physics-Based Adaptive Motion Primitives](https://mlsanity.com/paper/2102.04324/) (2021)
- [Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearranging](https://mlsanity.com/paper/2302.10446/) (2023)
- [Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning](https://mlsanity.com/paper/2309.02547/) (2023)
- [On Minimizing the Number of Running Buffers for Tabletop Rearrangement](https://mlsanity.com/paper/2105.06357/) (2021)
- [Efficient and High-quality Prehensile Rearrangement in Cluttered and Confined Spaces](https://mlsanity.com/paper/2110.02814/) (2021)
- [A Heuristically Assisted Deep Reinforcement Learning Approach for Network Slice Placement](https://mlsanity.com/paper/2105.06741/) (2021)
- [Minimizing Running Buffers for Tabletop Object Rearrangement: Complexity, Fast Algorithms, and Applications](https://mlsanity.com/paper/2304.01764/) (2023)

## Abstract (arXiv)

A combined task-level reinforcement learning and motion planning framework is proposed in this paper to address a multi-class in-rack test tube rearrangement problem. At the task level, the framework uses reinforcement learning to infer a sequence of swap actions while ignoring robotic motion details. At the motion level, the framework accepts the swapping action sequences inferred by task-level agents and plans the detailed robotic pick-and-place motion. The task and motion-level planning form a closed loop with the help of a condition set maintained for each rack slot, which allows the framework to perform replanning and effectively find solutions in the presence of low-level failures. Particularly for reinforcement learning, the framework leverages a distributed deep Q-learning structure with the Dueling Double Deep Q Network (D3QN) to acquire near-optimal policies and uses an A^star-based post-processing technique to amplify the collected training data. The D3QN and distributed learning help increase training efficiency. The post-processing helps complete unfinished action sequences and remove redundancy, thus making the training data more effective. We carry out both simulations and real-world studies to understand the performance of the proposed framework. The results verify the performance of the RL and post-processing and show that the closed-loop combination improves robustness. The framework is ready to incorporate various sensory feedback. The real-world studies also demonstrated the incorporation.