# Light Field Based 6DoF Tracking of Previously Unobserved Objects

arXiv: https://arxiv.org/abs/2512.13007 · HTML: https://mlsanity.com/paper/2512.13007/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [gsplat: An Open-Source Library for Gaussian Splatting](https://mlsanity.com/paper/2409.06765/) (2024)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025)
- [OnePose: One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2205.12257/) (2022)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation](https://mlsanity.com/paper/1912.00416/) (2019)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints](https://mlsanity.com/paper/1910.10750/) (2019)
- [IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing](https://mlsanity.com/paper/2412.15867/) (2024)
- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [Disentangling Light Fields for Super-Resolution and Disparity Estimation](https://mlsanity.com/paper/2202.10603/) (2022)
- [SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation](https://mlsanity.com/paper/2208.06661/) (2022)
- [UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections](https://mlsanity.com/paper/2312.13285/) (2023)
- [Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation](https://mlsanity.com/paper/2205.11047/) (2022)
- [Learning Sub-Pixel Disparity Distribution for Light Field Depth Estimation](https://mlsanity.com/paper/2208.09688/) (2022)
- [6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting](https://mlsanity.com/paper/2412.01543/) (2024)
- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [Uncertainty for SVBRDF Acquisition using Frequency Analysis](https://mlsanity.com/paper/2406.17774/) (2024)
- [Variational Disparity Estimation Framework for Plenoptic Image](https://mlsanity.com/paper/1804.06633/) (2018)

## Built on by (most cited first, 1 of 1)

- [Reflection-Robust 6DoF Object Tracking with Light Fields](https://mlsanity.com/paper/2610.04883/) (2026)

## Abstract (arXiv)

Object tracking is an important step in robotics and reautonomous driving pipelines, which has to generalize to previously unseen and complex objects. Existing high-performing methods often rely on pre-captured object views to build explicit reference models, which restricts them to a fixed set of known objects. However, such reference models can struggle with visually complex appearance, reducing the quality of tracking. In this work, we introduce an object tracking method based on light field images that does not depend on a pre-trained model, while being robust to complex visual behavior, such as reflections. We extract semantic and geometric features from light field inputs using vision foundation models and convert them into view-dependent Gaussian splats. These splats serve as a unified object representation, supporting differentiable rendering and pose optimization. We further introduce a light field object tracking dataset containing challenging reflective objects with precise ground truth poses. Experiments demonstrate that our method is competitive with state-of-the-art model-based trackers in these difficult cases, paving the way toward universal object tracking in robotic systems. Code/data available at https://github.com/nagonch/LiFT-6DoF.