mlsanityData to 7 Oct 2026

Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting

arXiv 2411.03555 ↗First posted 5 Nov 2024cs.CV cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

This paper introduces a method to enhance Interactive Imitation Learning (IIL) by extracting touch interaction points and tracking object movement from video demonstrations. The approach extends current IIL systems by providing robots with detailed knowledge of both where and how to interact with objects, particularly complex articulated ones like doors and drawers. By leveraging cutting-edge techniques such as 3D Gaussian Splatting and FoundationPose for tracking, this method allows robots to better understand and manipulate objects in dynamic environments. The research lays the foundation for more effective task learning and execution in autonomous robotic systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 28 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

19 of 28 references.

References · 19

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  6. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  7. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  8. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  9. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  10. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  11. 2024GS2Mesh: Surface Reconstruction from Gaussian Splatting via Novel Stereo Viewscited by 65
  12. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
7 more
  1. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  2. 2024Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Roboticscited by 39
  3. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  4. 2023UniTeam: Open Vocabulary Mobile Manipulation Challengecited by 15
  5. 2024Zero-shot Imitation Policy via Search in Demonstration Datasetcited by 13
  6. 2024Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challengecited by 9
  7. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
  2. 2026One-Shot Learning from Demonstration of Contact-Rich Robotic Manipulation by Identifying Physical Interactions
  3. 2026A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learning
  4. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
  5. 2024SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation
  6. 2023One-shot Imitation Learning via Interaction Warping
  7. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  8. 2026Demonstration Synthesis from a Single Scan via Gaussian Splatting for Visuomotor Policy Learning ↗