# SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations

arXiv: https://arxiv.org/abs/1903.06482 · HTML: https://mlsanity.com/paper/1903.06482/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2019.

## Built on by (most cited first, 30 of 37)

- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting](https://mlsanity.com/paper/2311.11700/) (2023)
- [NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM](https://mlsanity.com/paper/2302.03594/) (2023)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [vMAP: Vectorised Object Mapping for Neural Field SLAM](https://mlsanity.com/paper/2302.01838/) (2023)
- [Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation](https://mlsanity.com/paper/2105.07593/) (2021)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising](https://mlsanity.com/paper/2403.20034/) (2024)
- [ILabel: Interactive Neural Scene Labelling](https://mlsanity.com/paper/2111.14637/) (2021)
- [DNS SLAM: Dense Neural Semantic-Informed SLAM](https://mlsanity.com/paper/2312.00204/) (2023)
- [iDF-SLAM: End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking](https://mlsanity.com/paper/2209.07919/) (2022)
- [MoD-SLAM: Monocular Dense Mapping for Unbounded 3D Scene Reconstruction](https://mlsanity.com/paper/2402.03762/) (2024)
- [MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors](https://mlsanity.com/paper/2209.15153/) (2022)
- [Advances in Inference and Representation for Simultaneous Localization and Mapping](https://mlsanity.com/paper/2103.05041/) (2021)
- [MBA-VO: Motion Blur Aware Visual Odometry](https://mlsanity.com/paper/2103.13684/) (2021)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)
- [TextSLAM: Visual SLAM with Semantic Planar Text Features](https://mlsanity.com/paper/2305.10029/) (2023)
- [ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction](https://mlsanity.com/paper/2311.18068/) (2023)
- [End-to-End Egospheric Spatial Memory](https://mlsanity.com/paper/2102.07764/) (2021)
- [EGG-Fusion: Efficient 3D Reconstruction with Geometry-aware Gaussian Surfel on the Fly](https://mlsanity.com/paper/2512.01296/) (2025)
- [Bootstrapped Self-Supervised Training with Monocular Video for Semantic Segmentation and Depth Estimation](https://mlsanity.com/paper/2103.11031/) (2021)
- [NeB-SLAM: Neural Blocks-based Salable RGB-D SLAM for Unknown Scenes](https://mlsanity.com/paper/2405.15151/) (2024)
- [Unpaired Image Deraining Using Reward-Guided Self-Reinforcement Strategy](https://mlsanity.com/paper/2605.00719/) (2026)
- [A Front-End for Dense Monocular SLAM using a Learned Outlier Mask Prior](https://mlsanity.com/paper/2104.00562/) (2021)
- [Sparse Image based Navigation Architecture to Mitigate the need of precise Localization in Mobile Robots](https://mlsanity.com/paper/2203.15272/) (2022)
- [Differentiable SLAM Helps Deep Learning-based LiDAR Perception Tasks](https://mlsanity.com/paper/2309.09206/) (2023)

## Abstract (arXiv)

Systems which incrementally create 3D semantic maps from image sequences must store and update representations of both geometry and semantic entities. However, while there has been much work on the correct formulation for geometrical estimation, state-of-the-art systems usually rely on simple semantic representations which store and update independent label estimates for each surface element (depth pixels, surfels, or voxels). Spatial correlation is discarded, and fused label maps are incoherent and noisy. We introduce a new compact and optimisable semantic representation by training a variational auto-encoder that is conditioned on a colour image. Using this learned latent space, we can tackle semantic label fusion by jointly optimising the low-dimenional codes associated with each of a set of overlapping images, producing consistent fused label maps which preserve spatial correlation. We also show how this approach can be used within a monocular keyframe based semantic mapping system where a similar code approach is used for geometry. The probabilistic formulation allows a flexible formulation where we can jointly estimate motion, geometry and semantics in a unified optimisation.