# 3D Ground Truth Reconstruction from Multi-Camera Annotations Using UKF

arXiv: https://arxiv.org/abs/2511.17609 · HTML: https://mlsanity.com/paper/2511.17609/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Multiview Detection with Feature Perspective Transformation](https://mlsanity.com/paper/2007.07247/) (2020)
- [Deep Occlusion Reasoning for Multi-Camera Multi-Target Detection](https://mlsanity.com/paper/1704.05775/) (2017)
- [Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo](https://mlsanity.com/paper/2104.02273/) (2021)
- [Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images](https://mlsanity.com/paper/2109.05885/) (2021)
- [Deep Multi-camera People Detection](https://mlsanity.com/paper/1702.04593/) (2017)
- [LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2111.11892/) (2021)
- [EarlyBird: Early-Fusion for Multi-View Tracking in the Bird's Eye View](https://mlsanity.com/paper/2310.13350/) (2023)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation](https://mlsanity.com/paper/2404.02041/) (2024)
- [Multiple View Geometry Transformers for 3D Human Pose Estimation](https://mlsanity.com/paper/2311.10983/) (2023)
- [Visual Multi-Object Tracking with Re-Identification and Occlusion Handling using Labeled Random Finite Sets](https://mlsanity.com/paper/2407.08872/) (2024)
- [Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking](https://mlsanity.com/paper/2405.18606/) (2024)
- [3D Multi-Object Tracking Employing MS-GLMB Filter for Autonomous Driving](https://mlsanity.com/paper/2410.14977/) (2024)

## Abstract (arXiv)

Accurate 3D ground truth estimation is critical for applications such as autonomous navigation, surveillance, and robotics. This paper introduces a novel method that uses an Unscented Kalman Filter (UKF) to fuse 2D bounding box or pose keypoint ground truth annotations from multiple calibrated cameras into accurate 3D ground truth. By leveraging human-annotated ground-truth 2D, our proposed method, a multi-camera single-object tracking algorithm, transforms 2D image coordinates into robust 3D world coordinates through homography-based projection and UKF-based fusion. Our proposed algorithm processes multi-view data to estimate object positions and shapes while effectively handling challenges such as occlusion. We evaluate our method on the CMC, Wildtrack, and Panoptic datasets, demonstrating high accuracy in 3D localization compared to the available 3D ground truth. Unlike existing approaches that provide only ground-plane information, our method also outputs the full 3D shape of each object. Additionally, the algorithm offers a scalable and fully automatic solution for multi-camera systems using only 2D image annotations.