mlsanityData to 7 Oct 2026

Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds

arXiv 2210.07442 ↗First posted 14 Oct 2022cs.RO cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

We study how choices of input point cloud coordinate frames impact learning of manipulation skills from 3D point clouds. There exist a variety of coordinate frame choices to normalize captured robot-object-interaction point clouds. We find that different frames have a profound effect on agent learning performance, and the trend is similar across 3D backbone networks. In particular, the end-effector frame and the target-part frame achieve higher training efficiency than the commonly used world frame and robot-base frame in many tasks, intuitively because they provide helpful alignments among point clouds across time steps and thus can simplify visual module learning. Moreover, the well-performing frames vary across tasks, and some tasks may benefit from multiple frame candidates. We thus propose FrameMiners to adaptively select candidate frames and fuse their merits in a task-agnostic manner. Experimentally, FrameMiners achieves on-par or significantly higher performance than the best single-frame version on five fully physical manipulation tasks adapted from ManiSkill and OCRTOC. Without changing existing camera placements or adding extra cameras, point cloud frame mining can serve as a free lunch to improve 3D manipulation learning.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 55 of its 62 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

55 of 62 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  9. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  10. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
38 more
  1. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  2. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  4. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
  5. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  6. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  7. 2017Submanifold Sparse Convolutional Networkscited by 223
  8. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  9. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  10. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  11. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  12. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  13. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  14. 2021A System for General In-Hand Object Re-Orientationcited by 155
  15. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154
  16. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  17. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  18. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  19. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  20. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  21. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  22. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  23. 2020AMVNet: Assertion-based Multi-View Fusion Network for LiDAR Semantic Segmentationcited by 87
  24. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  25. 2017Intel RealSense Stereoscopic Depth Camerascited by 82
  26. 2019Rotation Invariant Convolutions for 3D Point Clouds Deep Learningcited by 80
  27. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  28. 2022Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptionscited by 78
  29. 2020A Rotation-Invariant Framework for Deep Point Cloud Analysiscited by 62
  30. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  31. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  32. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  33. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  34. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  35. 2020TORNADO-Net: mulTiview tOtal vaRiatioN semAntic segmentation with Diamond inceptiOn modulecited by 36
  36. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  37. 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28
  38. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 12; 2025: 4; 2026: 4.

31 papers.

Sort:

By area:

  1. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  2. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  3. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  4. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  5. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  6. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  7. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  8. 2023PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimationcited by 37
  9. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  10. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 27
  11. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  12. 2023ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentationcited by 18
19 more
  1. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  2. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learningcited by 16
  3. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  4. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  5. 2023On the Efficacy of 3D Point Cloud Reinforcement Learningcited by 9
  6. 2024Part-Guided 3D RL for Sim2Real Articulated Object Manipulationcited by 9
  7. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulationcited by 8
  8. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  9. 2026PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interactioncited by 6
  10. 2024DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learningcited by 5
  11. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  12. 2025Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalizationcited by 1
  13. 2026Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulationcited by 1
  14. 2026See like a Robot: Robot-Centric Pointmaps for VLA Modelscited by 1
  15. 2024Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policycited by 0
  16. 2024GSL-PCD: Improving Generalist-Specialist Learning with Point Cloud Feature-based Task Partitioningcited by 0
  17. 2025Deformable Cluster Manipulation via Whole-Arm Policy Learningcited by 0
  18. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  19. 2026The Embodiment Gap in Robot Foundation Modelscited by 0