# MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos

arXiv: https://arxiv.org/abs/2012.05360 · HTML: https://mlsanity.com/paper/2012.05360/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [Object-Compositional Neural Implicit Surfaces](https://mlsanity.com/paper/2207.09686/) (2022)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models](https://mlsanity.com/paper/2406.10224/) (2024)
- [ODAM: Object Detection, Association, and Mapping using Posed RGB Video](https://mlsanity.com/paper/2108.10165/) (2021)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detection](https://mlsanity.com/paper/2310.01401/) (2023)
- [ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Description](https://mlsanity.com/paper/2108.00355/) (2021)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022)
- [O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model](https://mlsanity.com/paper/2308.09591/) (2023)
- [SDFEst: Categorical Pose and Shape Estimation of Objects from RGB-D using Signed Distance Fields](https://mlsanity.com/paper/2207.04880/) (2022)
- [RGB-D-Based Categorical Object Pose and Shape Estimation: Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2301.08147/) (2023)
- [Category-level Neural Field for Reconstruction of Partially Observed Objects in Indoor Environment](https://mlsanity.com/paper/2406.08176/) (2024)
- [Scatter Points in Space: 3D Detection from Multi-view Monocular Images](https://mlsanity.com/paper/2208.14738/) (2022)
- [Constructing Metric-Semantic Maps using Floor Plan Priors for Long-Term Indoor Localization](https://mlsanity.com/paper/2303.10959/) (2023)
- [Temporally Consistent Object 6D Pose Estimation for Robot Control](https://mlsanity.com/paper/2605.02708/) (2026)

## Abstract (arXiv)

Semantic aware reconstruction is more advantageous than geometric-only reconstruction for future robotic and AR/VR applications because it represents not only where things are, but also what things are. Object-centric mapping is a task to build an object-level reconstruction where objects are separate and meaningful entities that convey both geometry and semantic information. In this paper, we present MOLTR, a solution to object-centric mapping using only monocular image sequences and camera poses. It is able to localise, track, and reconstruct multiple objects in an online fashion when an RGB camera captures a video of the surrounding. Given a new RGB frame, MOLTR firstly applies a monocular 3D detector to localise objects of interest and extract their shape codes that represent the object shapes in a learned embedding space. Detections are then merged to existing objects in the map after data association. Motion state (i.e. kinematics and the motion status) of each object is tracked by a multiple model Bayesian filter and object shape is progressively refined by fusing multiple shape code. We evaluate localisation, tracking, and reconstruction on benchmarking datasets for indoor and outdoor scenes, and show superior performance over previous approaches.