# SMORE: Simultaneous Map and Object REconstruction

arXiv: https://arxiv.org/abs/2406.13896 · HTML: https://mlsanity.com/paper/2406.13896/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [SUDS: Scalable Urban Dynamic Scenes](https://mlsanity.com/paper/2303.14536/) (2023)
- [NeuRAD: Neural Rendering for Autonomous Driving](https://mlsanity.com/paper/2311.15260/) (2023)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild](https://mlsanity.com/paper/2110.07604/) (2021)
- [KISS-ICP: In Defense of Point-to-Point ICP -- Simple, Accurate, and Robust Registration If Done the Right Way](https://mlsanity.com/paper/2209.15397/) (2022)
- [S-NeRF: Neural Radiance Fields for Street Views](https://mlsanity.com/paper/2303.00749/) (2023)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [Neural Scene Flow Prior](https://mlsanity.com/paper/2111.01253/) (2021)
- [NPMs: Neural Parametric Models for 3D Deformable Shapes](https://mlsanity.com/paper/2104.00702/) (2021)
- [Neural LiDAR Fields for Novel View Synthesis](https://mlsanity.com/paper/2305.01643/) (2023)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [Towards Long-Tailed 3D Detection](https://mlsanity.com/paper/2211.08691/) (2022)
- [CT-ICP: Real-time Elastic LiDAR Odometry with Loop Closure](https://mlsanity.com/paper/2109.12979/) (2021)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)
- [Dynamic SLAM: The Need For Speed](https://mlsanity.com/paper/2002.08584/) (2020)
- [Efficient Continuous-Time SLAM for 3D Lidar-Based Online Mapping](https://mlsanity.com/paper/1810.06802/) (2018)
- [Reconstructing Objects in-the-wild for Realistic Sensor Simulation](https://mlsanity.com/paper/2311.05602/) (2023)

## Built on by (most cited first, 3 of 3)

- [HiMo: High-Speed Objects Motion Compensation in Point Clouds](https://mlsanity.com/paper/2503.00803/) (2025)
- [DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method](https://mlsanity.com/paper/2508.17054/) (2025)
- [TeFlow: Enabling Multi-frame Supervision for Self-Supervised Feed-forward Scene Flow Estimation](https://mlsanity.com/paper/2602.19053/) (2026)

## Abstract (arXiv)

We present a method for dynamic surface reconstruction of large-scale urban scenes from LiDAR. Depth-based reconstructions tend to focus on small-scale objects or large-scale SLAM reconstructions that treat moving objects as outliers. We take a holistic perspective and optimize a compositional model of a dynamic scene that decomposes the world into rigidly-moving objects and the background. To achieve this, we take inspiration from recent novel view synthesis methods and frame the reconstruction problem as a global optimization over neural surfaces, ego poses, and object poses, which minimizes the error between composed spacetime surfaces and input LiDAR scans. In contrast to view synthesis methods, which typically minimize 2D errors with gradient descent, we minimize a 3D point-to-surface error by coordinate descent, which we decompose into registration and surface reconstruction steps. Each step can be handled well by off-the-shelf methods without any re-training. We analyze the surface reconstruction step for rolling-shutter LiDARs, and show that deskewing operations common in continuous time SLAM can be applied to dynamic objects as well, improving results over prior art by an order of magnitude. Beyond pursuing dynamic reconstruction as a goal in and of itself, we propose that such a system can be used to auto-label partially annotated sequences and produce ground truth annotation for hard-to-label problems such as depth completion and scene flow. Please see https://anishmadan23.github.io/smore/ for more visual results.