# ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection arXiv: https://arxiv.org/abs/2310.09069 ยท HTML: https://mlsanity.com/paper/2310.09069/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 34 references that are in the map. - First posted on arXiv 13 Oct 2023. ## Builds on (most cited first, 24 of 24) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016) - [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018) - [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020) - [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022) - [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018) - [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021) - [GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts](https://mlsanity.com/paper/2211.05272/) (2022) - [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022) - [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019) - [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021) - [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022) - [BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo](https://mlsanity.com/paper/2209.10248/) (2022) - [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021) - [GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF](https://mlsanity.com/paper/2210.06575/) (2022) - [UMPNet: Universal Manipulation Policy Network for Articulated Objects](https://mlsanity.com/paper/2109.05668/) (2021) - [AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions](https://mlsanity.com/paper/2112.00246/) (2021) - [End-to-End Affordance Learning for Robotic Manipulation](https://mlsanity.com/paper/2209.12941/) (2022) - [Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation](https://mlsanity.com/paper/2201.07779/) (2022) - [PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations](https://mlsanity.com/paper/2303.16958/) (2023) - [Robotic Grasping using Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04499/) (2020) - [Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps](https://mlsanity.com/paper/2009.12606/) (2020) - [SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning](https://mlsanity.com/paper/2111.14693/) (2021) - [On the Metrics for Evaluating Monocular Depth Estimation](https://mlsanity.com/paper/2302.10007/) (2023) ## Built on by (most cited first, 7 of 7) - [LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding](https://mlsanity.com/paper/2312.14074/) (2023) - [Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation](https://mlsanity.com/paper/2411.18623/) (2024) - [Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation](https://mlsanity.com/paper/2405.16486/) (2024) - [DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control](https://mlsanity.com/paper/2407.14758/) (2024) - [Watch Less, Feel More: Sim-to-Real RL for Generalizable Articulated Object Manipulation via Motion Adaptation and Impedance Control](https://mlsanity.com/paper/2502.14457/) (2025) - [AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation](https://mlsanity.com/paper/2606.17446/) (2026) - [MagicSim: A Unified Infrastructure for Executable Embodied Interaction](https://mlsanity.com/paper/2606.17511/) (2026) ## Abstract (arXiv) In the realm of future home-assistant robots, 3D articulated object manipulation is essential for enabling robots to interact with their environment. Many existing studies make use of 3D point clouds as the primary input for manipulation policies. However, this approach encounters challenges due to data sparsity and the significant cost associated with acquiring point cloud data, which can limit its practicality. In contrast, RGB images offer high-resolution observations using cost effective devices but lack spatial 3D geometric information. To overcome these limitations, we present a novel image-based robotic manipulation framework. This framework is designed to capture multiple perspectives of the target object and infer depth information to complement its geometry. Initially, the system employs an eye-on-hand RGB camera to capture an overall view of the target object. It predicts the initial depth map and a coarse affordance map. The affordance map indicates actionable areas on the object and serves as a constraint for selecting subsequent viewpoints. Based on the global visual prior, we adaptively identify the optimal next viewpoint for a detailed observation of the potential manipulation success area. We leverage geometric consistency to fuse the views, resulting in a refined depth map and a more precise affordance map for robot manipulation decisions. By comparing with prior works that adopt point clouds or RGB images as inputs, we demonstrate the effectiveness and practicality of our method. In the project webpage (https://sites.google.com/view/imagemanip), real world experiments further highlight the potential of our method for practical deployment.