# Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement

arXiv: https://arxiv.org/abs/2105.02857 · HTML: https://mlsanity.com/paper/2105.02857/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2021.

## Built on by (most cited first, 15 of 15)

- [CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation](https://mlsanity.com/paper/2109.09163/) (2021)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions](https://mlsanity.com/paper/2309.07510/) (2023)
- [Transporters with Visual Foresight for Solving Unseen Rearrangement Tasks](https://mlsanity.com/paper/2202.10765/) (2022)
- [Coarse-to-fine Q-attention with Tree Expansion](https://mlsanity.com/paper/2204.12471/) (2022)
- [GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation](https://mlsanity.com/paper/2503.09243/) (2025)
- [Efficient and High-quality Prehensile Rearrangement in Cluttered and Confined Spaces](https://mlsanity.com/paper/2110.02814/) (2021)
- [ORLA*: Mobile Manipulator-Based Object Rearrangement with Lazy A Star](https://mlsanity.com/paper/2309.13707/) (2023)
- [Planning for Manipulation among Movable Objects: Deciding Which Objects Go Where, in What Order, and How](https://mlsanity.com/paper/2303.13385/) (2023)
- [Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search](https://mlsanity.com/paper/2109.08973/) (2021)
- [Learning Dual-Arm Push and Grasp Synergy in Dense Clutter](https://mlsanity.com/paper/2412.04052/) (2024)
- [Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learning](https://mlsanity.com/paper/2508.19476/) (2025)
- [GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning](https://mlsanity.com/paper/2603.04158/) (2026)
- [Category-Adaptive Label Discovery and Noise Rejection for Multi-label Image Recognition with Partial Positive Labels](https://mlsanity.com/paper/2211.07846/) (2022)
- [Planning for Complex Non-prehensile Manipulation Among Movable Objects by Interleaving Multi-Agent Pathfinding and Physics-Based Simulation](https://mlsanity.com/paper/2303.13352/) (2023)

## Abstract (arXiv)

This paper considers the problem of retrieving an object from many tightly packed objects using a combination of robotic pushing and grasping actions. Object retrieval in dense clutter is an important skill for robots to operate in households and everyday environments effectively. The proposed solution, Visual Foresight Trees (VFT), intelligently rearranges the clutter surrounding a target object so that it can be grasped easily. Rearrangement with nested nonprehensile actions is challenging as it requires predicting complex object interactions in a combinatorially large configuration space of multiple objects. We first show that a deep neural network can be trained to accurately predict the poses of the packed objects when the robot pushes one of them. The predictive network provides visual foresight and is used in a tree search as a state transition function in the space of scene images. The tree search returns a sequence of consecutive push actions yielding the best arrangement of the clutter for grasping the target object. Experiments in simulation and using a real robot and objects show that the proposed approach outperforms model-free techniques as well as model-based myopic methods both in terms of success rates and the number of executed actions, on several challenging tasks. A video introducing VFT, with robot experiments, is accessible at https://youtu.be/7cL-hmgvyec. The full source code is available at https://github.com/arc-l/vft.