mlsanityData to 13 Oct 2026

FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

arXiv 2312.08344 ↗First posted 13 Dec 2023cs.CV cs.AI cs.ROcited by 415

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups. Our approach can be instantly applied at test-time to a novel object without fine-tuning, as long as its CAD model is given, or a small number of reference images are captured. We bridge the gap between these two setups with a neural implicit representation that allows for effective novel view synthesis, keeping the downstream pose estimation modules invariant under the same unified framework. Strong generalizability is achieved via large-scale synthetic training, aided by a large language model (LLM), a novel transformer-based architecture, and contrastive learning formulation. Extensive evaluation on multiple public datasets involving challenging scenarios and objects indicate our unified approach outperforms existing methods specialized for each task by a large margin. In addition, it even achieves comparable results to instance-level methods despite the reduced assumptions. Project page: https://nvlabs.github.io/FoundationPose/

Key facts. Cited by 415 papers in the map (data through 13 Oct 2026). Builds on 64 of its 79 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

64 of 79 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  8. 2017Mask R-CNNcited by 7,266
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,712
  10. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,706
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,529
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
38 more
  1. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,195
  2. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 979
  3. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 770
  4. 2020Implicit Geometric Regularization for Learning Shapescited by 708
  5. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 676
  6. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 605
  7. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 492
  8. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 440
  9. 2014Deep metric learning using Triplet networkcited by 422
  10. 2020PREDATOR: Registration of 3D Point Clouds with Low Overlapcited by 288
  11. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  12. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  13. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
  14. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
  15. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 232
  16. 2019Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimationcited by 218
  17. 2017T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objectscited by 210
  18. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  19. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 200
  20. 2020Shape Prior Deformation for Categorical 6D Object Pose and Size Estimationcited by 159
  21. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 156
  22. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 146
  23. 2021FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimationcited by 144
  24. 2020Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimationcited by 123
  25. 2022Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Imagescited by 120
  26. 2023OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Modelscited by 114
  27. 2021SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddingscited by 95
  28. 2019PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Trackingcited by 91
  29. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 89
  30. 2019LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimationcited by 88
  31. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  32. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 87
  33. 2023CNOS: A Strong Baseline for CAD-based Novel Object Segmentationcited by 86
  34. 2022FS6D: Few-Shot 6D Pose Estimation of Novel Objectscited by 81
  35. 2022OSOP: A Multi-Stage One Shot Object Pose Estimation Frameworkcited by 81
  36. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  37. 20196-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypointscited by 78
  38. 2022NeRF-Pose: A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimationcited by 57

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 52; 2025: 174; 2026: 188.

415 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 308
  2. 2025SAM 3D: 3Dfy Anything in Imagescited by 234
  3. 2025FoundationStereo: Zero-Shot Stereo Matchingcited by 170
  4. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 102
  5. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  6. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  7. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 68
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  9. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 52
  10. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 49
  11. 2026World Action Models: The Next Frontier in Embodied AIcited by 48
  12. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 47
38 more
  1. 2025Any6D: Model-free 6D Pose Estimation of Novel Objectscited by 47
  2. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  3. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 40
  4. 2024G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulationcited by 35
  5. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  6. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 34
  7. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 31
  8. 2025DexMachina: Functional Retargeting for Bimanual Dexterous Manipulationcited by 31
  9. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  10. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 28
  11. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 28
  12. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 27
  13. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 27
  14. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 26
  15. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 25
  16. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  17. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  18. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  19. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 24
  20. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 23
  21. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  22. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulationcited by 22
  23. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  24. 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 20
  25. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 18
  26. 2025Robot Learning from a Physical World Modelcited by 18
  27. 2024Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentationcited by 17
  28. 2025GCE-Pose: Global Context Enhancement for Category-level Object Pose Estimationcited by 17
  29. 2025CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-Worldcited by 17
  30. 2025One2Any: One-Reference 6D Pose Estimation for Any Objectcited by 17
  31. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 17
  32. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulationcited by 17
  33. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  34. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 16
  35. 2025SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Modelcited by 16
  36. 2024GS-Pose: Generalizable Segmentation-based 6D Object Pose Estimation with 3D Gaussian Splattingcited by 14
  37. 2024Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Modelscited by 14
  38. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 14

50 most cited of 415 All 415 (JSON)