# Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction

arXiv: https://arxiv.org/abs/2301.13244 · HTML: https://mlsanity.com/paper/2301.13244/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects](https://mlsanity.com/paper/1706.06629/) (2017)
- [SurfelWarp: Efficient Non-Volumetric Single View Dynamic Reconstruction](https://mlsanity.com/paper/1904.13073/) (2019)
- [OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction](https://mlsanity.com/paper/2203.07977/) (2022)
- [Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation](https://mlsanity.com/paper/2109.13910/) (2021)
- [Topology-Aware Non-Rigid Point Cloud Registration](https://mlsanity.com/paper/1811.07014/) (2018)
- [Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects](https://mlsanity.com/paper/1910.04953/) (2019)
- [Physics-based Scene-level Reasoning for Object Pose Estimation in Clutter](https://mlsanity.com/paper/1806.10457/) (2018)
- [Topology-Change-Aware Volumetric Fusion for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2007.06853/) (2020)
- [Scene-level Tracking and Reconstruction without Object Priors](https://mlsanity.com/paper/2210.03815/) (2022)

## Built on by (most cited first, 4 of 4)

- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs](https://mlsanity.com/paper/2309.15940/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Motion Blender Gaussian Splatting for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2503.09040/) (2025)

## Abstract (arXiv)

We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods.