mlsanityData to 7 Oct 2026

3D Implicit Transporter for Temporally Consistent Keypoint Discovery

arXiv 2309.05098 ↗First posted 10 Sep 2023cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Keypoint-based representation has proven advantageous in various visual and robotic tasks. However, the existing 2D and 3D methods for detecting keypoints mainly rely on geometric consistency to achieve spatial alignment, neglecting temporal consistency. To address this issue, the Transporter method was introduced for 2D data, which reconstructs the target frame from the source frame to incorporate both spatial and temporal information. However, the direct application of the Transporter to 3D point clouds is infeasible due to their structural differences from 2D images. Thus, we propose the first 3D version of the Transporter, which leverages hybrid 3D representation, cross attention, and implicit reconstruction. We apply this new learning system on 3D articulated objects and nonrigid animals (humans and rodents) and show that learned keypoints are spatio-temporally consistent. Additionally, we propose a closed-loop control strategy that utilizes the learned keypoints for 3D object manipulation and demonstrate its superior performance. Codes are available at https://github.com/zhongcl-thu/3D-Implicit-Transporter.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 10 Sep 2023.

Builds on

54 of 82 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  7. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  8. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  9. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  10. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  11. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  12. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
38 more
  1. 2020Neural Sparse Voxel Fieldscited by 959
  2. 2020Convolutional Occupancy Networkscited by 760
  3. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  4. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  5. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  6. 20163DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructionscited by 333
  7. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  8. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  9. 2020D3Feat: Joint Learning of Dense Detection and Description of 3D Local Featurescited by 175
  10. 2019Category-Level Articulated Object Pose Estimationcited by 158
  11. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  12. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  13. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  14. 2019Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapescited by 117
  15. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  16. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  17. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  18. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  19. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  20. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  21. 2021Lepard: Learning partial point cloud matching in rigid and deformable scenescited by 85
  22. 2021CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Cloudscited by 83
  23. 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
  24. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  25. 2019USIP: Unsupervised Stable Interest Point Detection from 3D Point Cloudscited by 70
  26. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  27. 2018Deep Part Induction from Articulated Object Pairscited by 66
  28. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  29. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  30. 2020CorrNet3D: Unsupervised End-to-end Learning of Dense Correspondence for 3D Point Cloudscited by 47
  31. 2022Disentangling Architecture and Training for Optical Flowcited by 32
  32. 2023LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDARcited by 31
  33. 2021Unsupervised Visual Attention and Invariance for Reinforcement Learningcited by 30
  34. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  35. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  36. 2021Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objectscited by 17
  37. 2014HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognitioncited by 15
  38. 2020UKPGAN: A General Self-Supervised Keypoint Detectorcited by 15

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 6; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
  2. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  3. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  4. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  5. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  6. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  7. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  8. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  9. 2025Infinite Mobility: Scalable High-Fidelity Synthesis of Articulated Objects via Procedural Generationcited by 17
  10. 2025GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objectscited by 17
  11. 2024FastMAC: Stochastic Spectral Sampling of Correspondence Graphcited by 16
  12. 2025PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Modelcited by 15
8 more
  1. 2025Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimationcited by 9
  2. 2026LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Modelscited by 8
  3. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  4. 2024GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulationcited by 4
  5. 2024SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapescited by 1
  6. 2024Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Losscited by 1
  7. 2024ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?cited by 1
  8. 2024Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priorscited by 1