# Visual-Inertial Object Detection and Mapping

arXiv: https://arxiv.org/abs/1806.08498 · HTML: https://mlsanity.com/paper/1806.08498/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2018.

## Built on by (most cited first, 5 of 5)

- [WorDepth: Variational Language Prior for Monocular Depth Estimation](https://mlsanity.com/paper/2404.03635/) (2024)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [Localization and Mapping using Instance-specific Mesh Models](https://mlsanity.com/paper/2103.04493/) (2021)
- [SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots](https://mlsanity.com/paper/2108.01262/) (2021)
- [Visual-Inertial Multi-Instance Dynamic SLAM with Object-level Relocalisation](https://mlsanity.com/paper/2208.04274/) (2022)

## Abstract (arXiv)

We present a method to populate an unknown environment with models of previously seen objects, placed in a Euclidean reference frame that is inferred causally and on-line using monocular video along with inertial sensors. The system we implement returns a sparse point cloud for the regions of the scene that are visible but not recognized as a previously seen object, and a detailed object model and its pose in the Euclidean frame otherwise. The system includes bottom-up and top-down components, whereby deep networks trained for detection provide likelihood scores for object hypotheses provided by a nonlinear filter, whose state serves as memory. Additional networks provide likelihood scores for edges, which complements detection networks trained to be invariant to small deformations. We test our algorithm on existing datasets, and also introduce the VISMA dataset, that provides ground truth pose, point-cloud map, and object models, along with time-stamped inertial measurements.