# GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos

arXiv: https://arxiv.org/abs/2412.02267 · HTML: https://mlsanity.com/paper/2412.02267/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Builds on (most cited first, 10 of 10)

- [No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images](https://mlsanity.com/paper/2410.24207/) (2024)
- [SparseGS: Sparse View Synthesis using 3D Gaussian Splatting](https://mlsanity.com/paper/2312.00206/) (2023)
- [NeRF-Pose: A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2203.04802/) (2022)
- [Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation](https://mlsanity.com/paper/2406.01349/) (2024)
- [GS-Pose: Generalizable Segmentation-based 6D Object Pose Estimation with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.10683/) (2024)
- [Object Gaussian for Monocular 6D Pose Estimation from Sparse Views](https://mlsanity.com/paper/2409.02581/) (2024)
- [Mobile AR Depth Estimation: Challenges & Prospects -- Extended Version](https://mlsanity.com/paper/2310.14437/) (2023)
- [Shape-IoU: More Accurate Metric considering Bounding Box Shape and Scale](https://mlsanity.com/paper/2312.17663/) (2023)
- [3D Reconstruction of Simple Objects from A Single View Silhouette Image](https://mlsanity.com/paper/1701.04752/) (2017)
- [UniQuadric: A SLAM Backend for Unknown Rigid Object 3D Tracking and Light-Weight Modeling](https://mlsanity.com/paper/2309.17036/) (2023)

## Abstract (arXiv)

Tracking the 6DoF pose of unknown objects in monocular RGB video sequences is crucial for robotic manipulation. However, existing approaches typically rely on accurate depth information, which is non-trivial to obtain in real-world scenarios. Although depth estimation algorithms can be employed, geometric inaccuracy can lead to failures in RGBD-based pose tracking methods. To address this challenge, we introduce GSGTrack, a novel RGB-based pose tracking framework that jointly optimizes geometry and pose. Specifically, we adopt 3D Gaussian Splatting to create an optimizable 3D representation, which is learned simultaneously with a graph-based geometry optimization to capture the object's appearance features and refine its geometry. However, the joint optimization process is susceptible to perturbations from noisy pose and geometry data. Thus, we propose an object silhouette loss to address the issue of pixel-wise loss being overly sensitive to pose noise during tracking. To mitigate the geometric ambiguities caused by inaccurate depth information, we propose a geometry-consistent image pair selection strategy, which filters out low-confidence pairs and ensures robust geometric optimization. Extensive experiments on the OnePose and HO3D datasets demonstrate the effectiveness of GSGTrack in both 6DoF pose tracking and object reconstruction.