# ManhattanSLAM: Robust Planar Tracking and Mapping Leveraging Mixture of Manhattan Frames

arXiv: https://arxiv.org/abs/2103.15068 · HTML: https://mlsanity.com/paper/2103.15068/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [PL-SLAM: a Stereo SLAM System through the Combination of Points and Line Segments](https://mlsanity.com/paper/1705.09479/) (2017)
- [SurfelMeshing: Online Surfel-Based Mesh Reconstruction](https://mlsanity.com/paper/1810.00729/) (2018)
- [InfiniTAM v3: A Framework for Large-Scale 3D Reconstruction with Loop Closure](https://mlsanity.com/paper/1708.00783/) (2017)
- [Structure-SLAM: Low-Drift Monocular SLAM in Indoor Environments](https://mlsanity.com/paper/2008.01963/) (2020)
- [Real-time Scalable Dense Surfel Mapping](https://mlsanity.com/paper/1909.04250/) (2019)
- [RGB-D SLAM with Structural Regularities](https://mlsanity.com/paper/2010.07997/) (2020)

## Built on by (most cited first, 18 of 18)

- [GeoGaussian: Geometry-aware Gaussian Splatting for Scene Rendering](https://mlsanity.com/paper/2403.11324/) (2024)
- [MG-SLAM: Structure Gaussian Splatting SLAM with Manhattan World Hypothesis](https://mlsanity.com/paper/2405.20031/) (2024)
- [Structure PLP-SLAM: Efficient Sparse Mapping and Localization using Point, Line and Plane for Monocular, RGB-D and Stereo Cameras](https://mlsanity.com/paper/2207.06058/) (2022)
- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [Q-SLAM: Quadric Representations for Monocular SLAM](https://mlsanity.com/paper/2403.08125/) (2024)
- [PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2110.11219/) (2021)
- [VIP-SLAM: An Efficient Tightly-Coupled RGB-D Visual Inertial Planar SLAM](https://mlsanity.com/paper/2207.01158/) (2022)
- [RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion](https://mlsanity.com/paper/2306.03584/) (2023)
- [Visual SLAM with Graph-Cut Optimized Multi-Plane Reconstruction](https://mlsanity.com/paper/2108.04281/) (2021)
- [E-Graph: Minimal Solution for Rigid Rotation with Extensibility Graphs](https://mlsanity.com/paper/2207.10008/) (2022)
- [Deep Single Image Camera Calibration by Heatmap Regression to Recover Fisheye Images Under Manhattan World Assumption](https://mlsanity.com/paper/2303.17166/) (2023)
- [MVGSR: Multi-View Consistency Gaussian Splatting for Robust Surface Reconstruction](https://mlsanity.com/paper/2503.08093/) (2025)
- [MSC-VO: Exploiting Manhattan and Structural Constraints for Visual Odometry](https://mlsanity.com/paper/2111.03408/) (2021)
- [PAS-SLAM: A Visual SLAM System for Planar Ambiguous Scenes](https://mlsanity.com/paper/2402.06131/) (2024)
- [EVOLIN Benchmark: Evaluation of Line Detection and Association](https://mlsanity.com/paper/2303.05162/) (2023)
- [Semantic Dense Reconstruction with Consistent Scene Segments](https://mlsanity.com/paper/2109.14821/) (2021)
- [RiemanLine: Riemannian Manifold Representation of 3D Lines for Factor Graph Optimization](https://mlsanity.com/paper/2508.04335/) (2025)
- [Does Dynamic-Point Filtering Help When Texture Is Scarce? A Controlled Study of ORB-SLAM2 Front-Ends in Synthetic Indoor Scenes](https://mlsanity.com/paper/2610.10564/) (2026)

## Abstract (arXiv)

In this paper, a robust RGB-D SLAM system is proposed to utilize the structural information in indoor scenes, allowing for accurate tracking and efficient dense mapping on a CPU. Prior works have used the Manhattan World (MW) assumption to estimate low-drift camera pose, in turn limiting the applications of such systems. This paper, in contrast, proposes a novel approach delivering robust tracking in MW and non-MW environments. We check orthogonal relations between planes to directly detect Manhattan Frames, modeling the scene as a Mixture of Manhattan Frames. For MW scenes, we decouple pose estimation and provide a novel drift-free rotation estimation based on Manhattan Frame observations. For translation estimation in MW scenes and full camera pose estimation in non-MW scenes, we make use of point, line and plane features for robust tracking in challenging scenes. %mapping Additionally, by exploiting plane features detected in each frame, we also propose an efficient surfel-based dense mapping strategy, which divides each image into planar and non-planar regions. Planar surfels are initialized directly from sparse planes in our map while non-planar surfels are built by extracting superpixels. We evaluate our method on public benchmarks for pose estimation, drift and reconstruction accuracy, achieving superior performance compared to other state-of-the-art methods. We will open-source our code in the future.