# Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation

arXiv: https://arxiv.org/abs/2103.10534 · HTML: https://mlsanity.com/paper/2103.10534/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 65 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory](https://mlsanity.com/paper/2008.10518/) (2020)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation](https://mlsanity.com/paper/2103.00946/) (2021)
- [Visual Identification of Articulated Object Parts](https://mlsanity.com/paper/2012.00284/) (2020)
- [FIESTA: Fast Incremental Euclidean Distance Fields for Online Motion Planning of Aerial Robots](https://mlsanity.com/paper/1903.02144/) (2019)
- [Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learning](https://mlsanity.com/paper/2101.05325/) (2021)
- [Robust and Adaptive Door Operation with a Mobile Robot](https://mlsanity.com/paper/1902.09051/) (2019)
- [Whole-Body MPC for a Dynamically Stable Mobile Manipulator](https://mlsanity.com/paper/1902.10415/) (2019)
- [An Efficient Optimal Planning and Control Framework For Quadrupedal Locomotion](https://mlsanity.com/paper/1609.09861/) (2016)
- [Real-Time Motion Planning of Legged Robots: A Model Predictive Control Approach](https://mlsanity.com/paper/1710.04029/) (2017)
- [Relaxed Logarithmic Barrier Function Based Model Predictive Control of Linear Systems](https://mlsanity.com/paper/1503.03314/) (2015)
- [Efficient Kinematic Planning for Mobile Manipulators with Non-holonomic Constraints Using Optimal Control](https://mlsanity.com/paper/1701.08051/) (2017)

## Built on by (most cited first, 30 of 44)

- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [UMPNet: Universal Manipulation Policy Network for Articulated Objects](https://mlsanity.com/paper/2109.05668/) (2021)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [OPD: Single-view 3D Openable Part Detection](https://mlsanity.com/paper/2203.16421/) (2022)
- [DribbleBot: Dynamic Legged Manipulation in the Wild](https://mlsanity.com/paper/2304.01159/) (2023)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [DITTO: Demonstration Imitation by Trajectory Transformation](https://mlsanity.com/paper/2403.15203/) (2024)
- [Pedipulate: Enabling Manipulation Skills using a Quadruped Robot's Leg](https://mlsanity.com/paper/2402.10837/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments](https://mlsanity.com/paper/2206.08737/) (2022)
- [Skill Transformer: A Monolithic Policy for Mobile Manipulation](https://mlsanity.com/paper/2308.09873/) (2023)
- [Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrations](https://mlsanity.com/paper/2203.02107/) (2022)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushing](https://mlsanity.com/paper/2411.07104/) (2024)
- [MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation](https://mlsanity.com/paper/2503.11081/) (2025)
- [Articulated 3D Scene Graphs for Open-World Mobile Manipulation](https://mlsanity.com/paper/2602.16356/) (2026)
- [Active-Perceptive Motion Generation for Mobile Manipulation](https://mlsanity.com/paper/2310.00433/) (2023)
- [MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation](https://mlsanity.com/paper/2308.03624/) (2023)
- [Towards Disturbance-Free Visual Mobile Manipulation](https://mlsanity.com/paper/2112.12612/) (2021)
- [Whole-body End-Effector Pose Tracking](https://mlsanity.com/paper/2409.16048/) (2024)
- [Predicting Motion Plans for Articulating Everyday Objects](https://mlsanity.com/paper/2303.01484/) (2023)
- [SM3: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects](https://mlsanity.com/paper/2401.09133/) (2024)

## Abstract (arXiv)

A kitchen assistant needs to operate human-scale objects, such as cabinets and ovens, in unmapped environments with dynamic obstacles. Autonomous interactions in such environments require integrating dexterous manipulation and fluid mobility. While mobile manipulators in different form factors provide an extended workspace, their real-world adoption has been limited. Executing a high-level task for general objects requires a perceptual understanding of the object as well as adaptive whole-body control among dynamic obstacles. In this paper, we propose a two-stage architecture for autonomous interaction with large articulated objects in unknown environments. The first stage, object-centric planner, only focuses on the object to provide an action-conditional sequence of states for manipulation using RGB-D data. The second stage, agent-centric planner, formulates the whole-body motion control as an optimal control problem that ensures safe tracking of the generated plan, even in scenes with moving obstacles. We show that the proposed pipeline can handle complex static and dynamic kitchen settings for both wheel-based and legged mobile manipulators. Compared to other agent-centric planners, our proposed planner achieves a higher success rate and a lower execution time. We also perform hardware tests on a legged mobile manipulator to interact with various articulated objects in a kitchen. For additional material, please check: www.pair.toronto.edu/articulated-mm/.