# Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects

arXiv: https://arxiv.org/abs/2006.15503 · HTML: https://mlsanity.com/paper/2006.15503/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 12 of 12)

- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances](https://mlsanity.com/paper/2108.04145/) (2021)
- [Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks](https://mlsanity.com/paper/2106.14070/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [ReorientBot: Learning Object Reorientation for Specific-Posed Placement](https://mlsanity.com/paper/2202.11092/) (2022)
- [RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control](https://mlsanity.com/paper/2210.11668/) (2022)
- [Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2105.14391/) (2021)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [Pick-and-place Manipulation Across Grippers Without Retraining: A Learning-optimization Diffusion Policy Approach](https://mlsanity.com/paper/2502.15613/) (2025)
- [Slot-Level Robotic Placement via Visual Imitation from Single Human Video](https://mlsanity.com/paper/2504.01959/) (2025)
- [AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement](https://mlsanity.com/paper/2604.10432/) (2026)
- [Hybrid Residual Reinforcement Learning for Contact-Rich Robotic Book Insertion](https://mlsanity.com/paper/2609.19962/) (2026)

## Abstract (arXiv)

Recent progress in robotic manipulation has dealt with the case of previously unknown objects in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work deals with pick-and-constrained placement of objects without access to geometric models. The objective is to pick an object and place it safely inside a desired goal region without any collisions, while minimizing the time and the sensing operations required to complete the task. An algorithmic framework is proposed for this purpose, which performs manipulation planning simultaneously over a conservative and an optimistic estimate of the object's volume. The conservative estimate ensures that the manipulation is safe while the optimistic estimate guides the sensor-based manipulation process when no solution can be found for the conservative estimate. To maintain these estimates and dynamically update them during manipulation, objects are represented by a simple volumetric representation, which stores sets of occupied and unseen voxels. The effectiveness of the proposed approach is demonstrated by developing a robotic system that picks a previously unseen object from a table-top and places it in a constrained space. The system comprises of a dual-arm manipulator with heterogeneous end-effectors and leverages hand-offs as a re-grasping strategy. Real-world experiments show that straightforward pick-sense-and-place alternatives frequently fail to solve pick-and-constrained placement problems. The proposed pipeline, however, achieves more than 95% success rate and faster execution times as evaluated over multiple physical experiments.