# Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation

arXiv: https://arxiv.org/abs/2105.07593 · HTML: https://mlsanity.com/paper/2105.07593/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 70 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 30 of 46)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning](https://mlsanity.com/paper/1709.06841/) (2017)
- [Particle Filter Networks with Application to Visual Localization](https://mlsanity.com/paper/1805.08975/) (2018)
- [Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors](https://mlsanity.com/paper/1805.11122/) (2018)
- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Path Planning using Neural A* Search](https://mlsanity.com/paper/2009.07476/) (2020)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Backprop KF: Learning Discriminative Deterministic State Estimators](https://mlsanity.com/paper/1605.07148/) (2016)

## Built on by (most cited first, 30 of 34)

- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Is Mapping Necessary for Realistic PointGoal Navigation?](https://mlsanity.com/paper/2206.00997/) (2022)
- [Uncertainty-driven Planner for Exploration and Navigation](https://mlsanity.com/paper/2202.11907/) (2022)
- [An overview of differentiable particle filters for data-adaptive sequential Bayesian inference](https://mlsanity.com/paper/2302.09639/) (2023)
- [Multi-Object Navigation with dynamically learned neural implicit representations](https://mlsanity.com/paper/2210.05129/) (2022)
- [Learning Active Camera for Multi-Object Navigation](https://mlsanity.com/paper/2210.07505/) (2022)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Ask4Help: Learning to Leverage an Expert for Embodied Tasks](https://mlsanity.com/paper/2211.09960/) (2022)
- [Differentiable Particle Filters through Conditional Normalizing Flow](https://mlsanity.com/paper/2107.00488/) (2021)
- [Vision and Language Navigation in the Real World via Online Visual Language Mapping](https://mlsanity.com/paper/2310.10822/) (2023)
- [RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting](https://mlsanity.com/paper/2404.19706/) (2024)
- [Compositional Learning-based Planning for Vision POMDPs](https://mlsanity.com/paper/2112.09456/) (2021)
- [RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion](https://mlsanity.com/paper/2306.03584/) (2023)
- [Modality-invariant Visual Odometry for Embodied Vision](https://mlsanity.com/paper/2305.00348/) (2023)
- [Spot the Difference: A Novel Task for Embodied Agents in Changing Environments](https://mlsanity.com/paper/2204.08502/) (2022)
- [LoRD: Adapting Differentiable Driving Policies to Distribution Shifts](https://mlsanity.com/paper/2410.09681/) (2024)
- [Monocular visual simultaneous localization and mapping: (r)evolution from geometry to deep learning-based pipelines](https://mlsanity.com/paper/2503.02955/) (2025)
- [Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation](https://mlsanity.com/paper/2504.08806/) (2025)
- [Self-Supervised Domain Adaptation for Visual Navigation with Global Map Consistency](https://mlsanity.com/paper/2110.07184/) (2021)
- [ViewBirdiformer: Learning to recover ground-plane crowd trajectories and ego-motion from a single ego-centric view](https://mlsanity.com/paper/2210.06332/) (2022)
- [MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents](https://mlsanity.com/paper/2211.15992/) (2022)
- [A General Purpose Supervisory Signal for Embodied Agents](https://mlsanity.com/paper/2212.01186/) (2022)
- [Resampling-free Particle Filters in High-dimensions](https://mlsanity.com/paper/2404.13698/) (2024)
- [Sparse Image based Navigation Architecture to Mitigate the need of precise Localization in Mobile Robots](https://mlsanity.com/paper/2203.15272/) (2022)
- [Unsupervised Visual Odometry and Action Integration for PointGoal Navigation in Indoor Environment](https://mlsanity.com/paper/2210.00413/) (2022)
- [Permutation Invariant Learning with High-Dimensional Particle Filters](https://mlsanity.com/paper/2410.22695/) (2024)
- [Belief-Conditioned One-Step Diffusion: Real-Time Trajectory Planning with Just-Enough Sensing](https://mlsanity.com/paper/2508.12166/) (2025)

## Abstract (arXiv)

Simultaneous localization and mapping (SLAM) remains challenging for a number of downstream applications, such as visual robot navigation, because of rapid turns, featureless walls, and poor camera quality. We introduce the Differentiable SLAM Network (SLAM-net) along with a navigation architecture to enable planar robot navigation in previously unseen indoor environments. SLAM-net encodes a particle filter based SLAM algorithm in a differentiable computation graph, and learns task-oriented neural network components by backpropagating through the SLAM algorithm. Because it can optimize all model components jointly for the end-objective, SLAM-net learns to be robust in challenging conditions. We run experiments in the Habitat platform with different real-world RGB and RGB-D datasets. SLAM-net significantly outperforms the widely adapted ORB-SLAM in noisy conditions. Our navigation architecture with SLAM-net improves the state-of-the-art for the Habitat Challenge 2020 PointNav task by a large margin (37% to 64% success). Project website: http://sites.google.com/view/slamnet