mlsanityData to 7 Oct 2026

BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

arXiv 2303.14158 ↗First posted 24 Mar 2023cs.CV cs.AI cs.GRcited by 144

Not on the map. Why?

Abstract (arXiv)

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches. Project page: https://bundlesdf.github.io

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 67 of its 89 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

67 of 89 references.

References · 67

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  4. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  5. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  6. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  8. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  9. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  10. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
38 more
  1. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  2. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  3. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  4. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  5. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  6. 20163DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructionscited by 333
  7. 2021Neural RGB-D Surface Reconstructioncited by 312
  8. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  9. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  10. 2020TEASER: Fast and Certifiable Point Cloud Registrationcited by 274
  11. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  12. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  13. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 198
  14. 2019Implicit 3D Orientation Learning for 6D Object Detection from RGB Imagescited by 170
  15. 2020Shape Prior Deformation for Categorical 6D Object Pose and Size Estimationcited by 158
  16. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 155
  17. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  18. 2021GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimatorcited by 148
  19. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  20. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  21. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  22. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  23. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  24. 2020Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimationcited by 123
  25. 2022Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Imagescited by 120
  26. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  27. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  28. 2018Fusion++: Volumetric Object-Level SLAMcited by 97
  29. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  30. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  31. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  32. 2019LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimationcited by 88
  33. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 87
  34. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  35. 2021CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Cloudscited by 83
  36. 2022FS6D: Few-Shot 6D Pose Estimation of Novel Objectscited by 81
  37. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  38. 20196-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypointscited by 78

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 40; 2025: 50; 2026: 33.

144 papers.

Sort:

By area:

  1. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  3. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  4. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  5. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  6. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  7. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  8. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  9. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 47
  10. 2025Any6D: Model-free 6D Pose Estimation of Novel Objectscited by 47
  11. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  12. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 39
38 more
  1. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  2. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  3. 2023Open-vocabulary object 6D pose estimationcited by 29
  4. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  5. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 25
  6. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  7. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 22
  8. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  9. 2024Neural Fields in Robotics: A Surveycited by 21
  10. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 21
  11. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 18
  12. 2025Robot Learning from a Physical World Modelcited by 18
  13. 2024NeRFs in Robotics: A Surveycited by 16
  14. 2023Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environmentscited by 15
  15. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  16. 2024Learning rigid-body simulators over implicit shapes for large-scale scenes and visioncited by 12
  17. 2023Diff-DOPE: Differentiable Deep Object Pose Estimationcited by 11
  18. 2025Scalable Unseen Objects 6-DoF Absolute Pose Estimation with Robotic Integrationcited by 11
  19. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  20. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  21. 2024OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimationcited by 10
  22. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  23. 2024Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphscited by 10
  24. 20246DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splattingcited by 10
  25. 20256D Object Pose Tracking in Internet Videos for Robotic Manipulationcited by 10
  26. 2025GoTrack: Generic 6DoF Object Pose Refinement and Trackingcited by 10
  27. 2024CRISP: Object Pose and Shape Estimation with Test-Time Adaptationcited by 9
  28. 2025UA-Pose: Uncertainty-Aware 6D Object Pose Estimation and Online Object Completion with Partial Referencescited by 9
  29. 2025ScrewSplat: An End-to-End Method for Articulated Object Recognitioncited by 9
  30. 20253D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Spacecited by 9
  31. 2024RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Renderingcited by 8
  32. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  33. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  34. 2023Color-NeuS: Reconstructing Neural Implicit Surfaces with Colorcited by 7
  35. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  36. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  37. 2026FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videoscited by 7
  38. 2024SMURF: Continuous Dynamics for Motion-Deblurring Radiance Fieldscited by 6

50 most cited of 144 All 144 (JSON)