mlsanityData to 14 Oct 2026

Tracking by 3D Model Estimation of Unknown Objects in Videos

arXiv 2304.06419 ↗First posted 13 Apr 2023cs.CV cs.GRcited by 5

Not on the map. Why?

Abstract (arXiv)

Most model-free visual object tracking methods formulate the tracking task as object location estimation given by a 2D segmentation or a bounding box in each video frame. We argue that this representation is limited and instead propose to guide and improve 2D tracking with an explicit object representation, namely the textured 3D shape and 6DoF pose in each video frame. Our representation tackles a complex long-term dense correspondence problem between all 3D points on the object for all video frames, including frames where some points are invisible. To achieve that, the estimation is driven by re-rendering the input video frames as well as possible through differentiable rendering, which has not been used for tracking before. The proposed optimization minimizes a novel loss function to estimate the best 3D shape, texture, and 6DoF pose. We improve the state-of-the-art in 2D segmentation tracking on three different datasets with mostly rigid objects.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

37 of 60 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  3. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  4. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  5. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  6. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  7. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
  8. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 330
  9. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  10. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  11. 2020ARCH: Animatable Reconstruction of Clothed Humanscited by 257
  12. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
25 more
  1. 2016One-Shot Video Object Segmentationcited by 235
  2. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  3. 2021Pixel-Perfect Structure-from-Motion with Featuremetric Refinementcited by 200
  4. 2020BOP Challenge 2020 on 6D Object Localizationcited by 193
  5. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 180
  6. 2020EPOS: Estimating 6D Pose of Objects with Symmetriescited by 148
  7. 2015A Novel Performance Evaluation Methodology for Single-Target Trackerscited by 113
  8. 2021ShaRF: Shape-conditioned Radiance Fields from a Single Viewcited by 92
  9. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  10. 2020Alpha-Refine: Boosting Tracking Performance by Precise Bounding Box Estimationcited by 81
  11. 2020Self6D: Self-Supervised Monocular 6D Object Pose Estimationcited by 79
  12. 2019D3S -- A Discriminative Single Shot Segmentation Trackercited by 69
  13. 2021Multi-view 3D Reconstruction with Transformercited by 52
  14. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  15. 2022Coupled Iterative Refinement for 6D Multi-Object Pose Estimationcited by 48
  16. 2018Object Tracking by Reconstruction with View-Specific Discriminative Correlation Filterscited by 33
  17. 2019CDTB: A Color and Depth Visual Object Tracking Dataset and Benchmarkcited by 32
  18. 2022CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wildcited by 28
  19. 2022In-Hand 3D Object Scanning from an RGB Sequencecited by 27
  20. 2022PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6 DoF Trackingcited by 15
  21. 2021Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videoscited by 11
  22. 2021Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objectscited by 10
  23. 2022DirectTracker: 3D Multi-Object Tracking Using Direct Image Alignment and Photometric Bundle Adjustmentcited by 6
  24. 2019Motion-Nets: 6D Tracking of Unknown Objects in Unseen Environments using RGBcited by 3
  25. 2019Visual Coin-Tracking: Tracking of Planar Double-Sided Objectscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  2. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  3. 2025PMGS: Reconstruction of Projectile Motion Across Large Spatiotemporal Spans via 3D Gaussian Splattingcited by 5
  4. 2024Dense Matchers for Dense Trackingcited by 1
  5. 2026Tracking the Unseen: An Occlusion-Robust Framework for Target Tracking Under Full and Long-Term Occlusioncited by 0