mlsanityData to 7 Oct 2026

Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly

arXiv 2110.04450 ↗First posted 9 Oct 2021cs.RO cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Studies in robot teleoperation have been centered around action specifications -- from continuous joint control to discrete end-effector pose control. However, these robot-centric interfaces often require skilled operators with extensive robotics expertise. To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects. As a result, a user can perform teleoperation without any expert knowledge of the robot hardware. To achieve this goal, we utilize a category-agnostic scene-completion algorithm that translates the real-world workspace (with unknown objects) into a manipulable virtual scene representation and an action-snapping algorithm that refines the user input before generating the robot's action plan. To train the algorithms, we procedurally generated a large-scale, diverse kit-assembly dataset that contains object-kit pairs that mimic real-world object-kitting tasks. Our experiments in simulation and on a real-world system demonstrate that our framework improves both the efficiency and success rate for 6DoF kit-assembly tasks. A user study demonstrates that SEaT framework participants achieve a higher task success rate and report a lower subjective workload compared to an alternative robot-centric interface. Video can be found at https://www.youtube.com/watch?v=-NdR3mkPbQQ .

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

23 of 53 references.

References · 23

  1. 2017Mask R-CNNcited by 7,259
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  4. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  5. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  6. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  7. 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
  8. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  9. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  10. 2014Deep Convolutional Neural Fields for Depth Estimation from a Single Imagecited by 106
  11. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  12. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
11 more
  1. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  2. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 50
  3. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  4. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
  5. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  6. 2016High precision grasp pose detection in dense cluttercited by 33
  7. 2018RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Cluttercited by 21
  8. 2018Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Imagescited by 11
  9. 2018In-Hand Manipulation via Motion Conescited by 9
  10. 2015Semantic Pose using Deep Networks Trained on Synthetic RGB-Dcited by 6
  11. 2021Kit-Net: Self-Supervised Learning to Kit Novel 3D Objects into Novel 3D Cavitiescited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2.

6 papers.

  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  3. 2024JUICER: Data-Efficient Imitation Learning for Robotic Assemblycited by 36
  4. 2023Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assemblycited by 27
  5. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
  6. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6