mlsanityData to 7 Oct 2026

Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation

arXiv 2105.07593 ↗First posted 17 May 2021cs.CV cs.AI cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Simultaneous localization and mapping (SLAM) remains challenging for a number of downstream applications, such as visual robot navigation, because of rapid turns, featureless walls, and poor camera quality. We introduce the Differentiable SLAM Network (SLAM-net) along with a navigation architecture to enable planar robot navigation in previously unseen indoor environments. SLAM-net encodes a particle filter based SLAM algorithm in a differentiable computation graph, and learns task-oriented neural network components by backpropagating through the SLAM algorithm. Because it can optimize all model components jointly for the end-objective, SLAM-net learns to be robust in challenging conditions. We run experiments in the Habitat platform with different real-world RGB and RGB-D datasets. SLAM-net significantly outperforms the widely adapted ORB-SLAM in noisy conditions. Our navigation architecture with SLAM-net improves the state-of-the-art for the Habitat Challenge 2020 PointNav task by a large margin (37% to 64% success). Project website: http://sites.google.com/view/slamnet

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

46 of 70 references.

References · 46

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2015Spatial Transformer Networkscited by 1,423
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  7. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 801
  8. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  11. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 405
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
34 more
  1. 2017Sparsity Invariant CNNscited by 325
  2. 2020Learning to Explore using Active Neural SLAMcited by 308
  3. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  4. 2020Neural Topological SLAM for Visual Navigationcited by 193
  5. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  6. 2018BA-Net: Dense Bundle Adjustment Networkcited by 173
  7. 2016Value Iteration Networkscited by 158
  8. 2017CNN-SLAM: Real-time dense monocular SLAM with learned depth predictioncited by 152
  9. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  10. 2017Value Prediction Networkcited by 117
  11. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  12. 2017UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learningcited by 95
  13. 2018Particle Filter Networks with Application to Visual Localizationcited by 64
  14. 2021Differentiable Particle Filtering via Entropy-Regularized Optimal Transportcited by 59
  15. 2018Differentiable Particle Filters: End-to-End Learning with Algorithmic Priorscited by 58
  16. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  17. 2020Path Planning using Neural A* Searchcited by 51
  18. 2016Backprop KF: Learning Discriminative Deterministic State Estimatorscited by 47
  19. 2019PyRobot: An Open-source Robotics Framework for Research and Benchmarkingcited by 39
  20. 2019SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representationscited by 37
  21. 2019Particle Filter Recurrent Neural Networkscited by 36
  22. 2020A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligencecited by 35
  23. 2018Learning to Search with MCTSnetscited by 32
  24. 2019Differentiable Gaussian Process Motion Planningcited by 27
  25. 2019gradSLAM: Automagically differentiable SLAMcited by 27
  26. 2020Discriminative Particle Filter Reinforcement Learning for Complex Partial Observationscited by 27
  27. 2019Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?cited by 25
  28. 2019Benchmarking Classic and Learned Navigation in Complex 3D Environmentscited by 24
  29. 2019Differentiable Algorithm Networks for Composable Robot Learningcited by 24
  30. 2020Towards Differentiable Resamplingcited by 19
  31. 2017Path Integral Networks: End-to-End Differentiable Optimal Controlcited by 16
  32. 2020Infinite-Horizon Differentiable Model Predictive Controlcited by 16
  33. 2017TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learningcited by 11
  34. 2020Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localizationcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 14; 2023: 4; 2024: 7; 2025: 3.

34 papers.

Sort:

By area:

  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  3. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  4. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehiclescited by 36
  5. 2022Is Mapping Necessary for Realistic PointGoal Navigation?cited by 32
  6. 2022Uncertainty-driven Planner for Exploration and Navigationcited by 31
  7. 2023An overview of differentiable particle filters for data-adaptive sequential Bayesian inferencecited by 23
  8. 2022Multi-Object Navigation with dynamically learned neural implicit representationscited by 21
  9. 2022Learning Active Camera for Multi-Object Navigationcited by 20
  10. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  11. 2022Pay Self-Attention to Audio-Visual Navigationcited by 17
  12. 2021Differentiable Particle Filters through Conditional Normalizing Flowcited by 15
22 more
  1. 2023Vision and Language Navigation in the Real World via Online Visual Language Mappingcited by 11
  2. 2024RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splattingcited by 10
  3. 2021Compositional Learning-based Planning for Vision POMDPscited by 8
  4. 2023RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completioncited by 7
  5. 2023Modality-invariant Visual Odometry for Embodied Visioncited by 6
  6. 2022Spot the Difference: A Novel Task for Embodied Agents in Changing Environmentscited by 5
  7. 2024LoRD: Adapting Differentiable Driving Policies to Distribution Shiftscited by 4
  8. 2025Monocular visual simultaneous localization and mapping: (r)evolution from geometry to deep learning-based pipelinescited by 4
  9. 2025Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigationcited by 4
  10. 2021Self-Supervised Domain Adaptation for Visual Navigation with Global Map Consistencycited by 3
  11. 2022ViewBirdiformer: Learning to recover ground-plane crowd trajectories and ego-motion from a single ego-centric viewcited by 3
  12. 2022MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agentscited by 3
  13. 2022A General Purpose Supervisory Signal for Embodied Agentscited by 3
  14. 2024Resampling-free Particle Filters in High-dimensionscited by 2
  15. 2022Sparse Image based Navigation Architecture to Mitigate the need of precise Localization in Mobile Robotscited by 1
  16. 2022Unsupervised Visual Odometry and Action Integration for PointGoal Navigation in Indoor Environmentcited by 1
  17. 2024Permutation Invariant Learning with High-Dimensional Particle Filterscited by 1
  18. 2025Belief-Conditioned One-Step Diffusion: Real-Time Trajectory Planning with Just-Enough Sensingcited by 1
  19. 2021Realistic PointGoal Navigation via Auxiliary Losses and Information Bottleneckcited by 0
  20. 2022Structured Exploration Through Instruction Enhancement for Object Navigationcited by 0
  21. 2024Inverse Particle Filtercited by 0
  22. 2024MPVO: Motion-Prior based Visual Odometry for PointGoal Navigationcited by 0