mlsanityData to 14 Oct 2026

Part-Guided 3D RL for Sim2Real Articulated Object Manipulation

arXiv 2404.17302 ↗First posted 26 Apr 2024cs.RO cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Manipulating unseen articulated objects through visual feedback is a critical but challenging task for real robots. Existing learning-based solutions mainly focus on visual affordance learning or other pre-trained visual models to guide manipulation policies, which face challenges for novel instances in real-world scenarios. In this paper, we propose a novel part-guided 3D RL framework, which can learn to manipulate articulated objects without demonstrations. We combine the strengths of 2D segmentation and 3D RL to improve the efficiency of RL policy training. To improve the stability of the policy on real robots, we design a Frame-consistent Uncertainty-aware Sampling (FUS) strategy to get a condensed and hierarchical 3D representation. In addition, a single versatile RL policy can be trained on multiple articulated object manipulation tasks simultaneously in simulation and shows great generalizability to novel categories and instances. Experimental results demonstrate the effectiveness of our framework in both simulation and real-world settings. Our code is available at https://github.com/THU-VCLab/Part-Guided-3D-RL-for-Sim2Real-Articulated-Object-Manipulation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

31 of 38 references.

References · 31

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  11. 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 280
  12. 2022Masked Visual Pre-training for Motor Controlcited by 248
19 more
  1. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  2. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  3. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 156
  4. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 139
  5. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  6. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  7. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  8. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 80
  9. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 65
  10. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 57
  11. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 57
  12. 2019S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenescited by 51
  13. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 51
  14. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 47
  15. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 32
  16. 2019DoorGym: A Scalable Door Opening Environment And Baseline Agentcited by 26
  17. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 21
  18. 2022Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulationcited by 16
  19. 2023Sim2Real2: Actively Building Explicit Physics Model for Precise Articulated Object Manipulationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2025PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understandingcited by 26
  2. 2024Diffusion-Based Depth Inpainting for Transparent and Reflective Objectscited by 9
  3. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  4. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 6
  5. 2026Energy-Aware Reinforcement Learning for Robotic Manipulation of Articulated Components in Infrastructure Operation and Maintenancecited by 3
  6. 2025FLEX: A Framework for Learning Robot-Agnostic Force-based Skills Involving Sustained Contact Object Manipulationcited by 2
  7. 2024Efficient 3D Perception on Multi-Sweep Point Cloud with Gumbel Spatial Pruningcited by 1
  8. 2025Data-Efficient Multitask DAggercited by 1
  9. 2025ArtReg: Visuo-Tactile based Pose Tracking and Manipulation of Unseen Articulated Objectscited by 0