# Planning for Complex Non-prehensile Manipulation Among Movable Objects by Interleaving Multi-Agent Pathfinding and Physics-Based Simulation

arXiv: https://arxiv.org/abs/2303.13352 · HTML: https://mlsanity.com/paper/2303.13352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 8 of 8)

- [Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement](https://mlsanity.com/paper/2105.02857/) (2021)
- [Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search](https://mlsanity.com/paper/2101.12241/) (2021)
- [Manipulation Planning Among Movable Obstacles Using Physics-Based Adaptive Motion Primitives](https://mlsanity.com/paper/2102.04324/) (2021)
- [Persistent Homology for Effective Non-Prehensile Manipulation](https://mlsanity.com/paper/2202.02937/) (2022)
- [Efficient Obstacle Rearrangement for Object Manipulation Tasks in Cluttered Environments](https://mlsanity.com/paper/1902.06907/) (2019)
- [Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints](https://mlsanity.com/paper/2005.09127/) (2020)
- [Planning with Selective Physics-based Simulation for Manipulation Among Movable Objects](https://mlsanity.com/paper/2003.06743/) (2020)
- [PERCH 2.0 : Fast and Accurate GPU-based Perception via Search for Object Pose Estimation](https://mlsanity.com/paper/2008.00326/) (2020)

## Abstract (arXiv)

Real-world manipulation problems in heavy clutter require robots to reason about potential contacts with objects in the environment. We focus on pick-and-place style tasks to retrieve a target object from a shelf where some `movable' objects must be rearranged in order to solve the task. In particular, our motivation is to allow the robot to reason over and consider non-prehensile rearrangement actions that lead to complex robot-object and object-object interactions where multiple objects might be moved by the robot simultaneously, and objects might tilt, lean on each other, or topple. To support this, we query a physics-based simulator to forward simulate these interaction dynamics which makes action evaluation during planning computationally very expensive. To make the planner tractable, we establish a connection between the domain of Manipulation Among Movable Objects and Multi-Agent Pathfinding that lets us decompose the problem into two phases our M4M algorithm iterates over. First we solve a multi-agent planning problem that reasons about the configurations of movable objects but does not forward simulate a physics model. Next, an arm motion planning problem is solved that uses a physics-based simulator but does not search over possible configurations of movable objects. We run simulated and real-world experiments with the PR2 robot and compare against relevant baseline algorithms. Our results highlight that M4M generates complex 3D interactions, and solves at least twice as many problems as the baselines with competitive performance.