# Differentiable SLAM Helps Deep Learning-based LiDAR Perception Tasks

arXiv: https://arxiv.org/abs/2309.09206 · HTML: https://mlsanity.com/paper/2309.09206/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [Learning to Generate Realistic LiDAR Point Clouds](https://mlsanity.com/paper/2209.03954/) (2022)
- [SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds](https://mlsanity.com/paper/2004.02774/) (2020)
- [Deep Generative Modeling of LiDAR Data](https://mlsanity.com/paper/1812.01180/) (2018)
- [Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data](https://mlsanity.com/paper/2105.08971/) (2021)
- [Patchwork++: Fast and Robust Ground Segmentation Solving Partial Under-Segmentation Using 3D Point Cloud](https://mlsanity.com/paper/2207.11919/) (2022)
- [SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations](https://mlsanity.com/paper/1903.06482/) (2019)
- [Patchwork: Concentric Zone-based Region-wise Ground Segmentation with Ground Likelihood Estimation Using a 3D LiDAR Sensor](https://mlsanity.com/paper/2108.05560/) (2021)
- [Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation](https://mlsanity.com/paper/2205.01823/) (2022)
- [gradSLAM: Automagically differentiable SLAM](https://mlsanity.com/paper/1910.10672/) (2019)
- [Learning to Drop Points for LiDAR Scan Synthesis](https://mlsanity.com/paper/2102.11952/) (2021)
- [Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data](https://mlsanity.com/paper/2210.11750/) (2022)
- [Differentiable Factor Graph Optimization for Learning Smoothers](https://mlsanity.com/paper/2105.08257/) (2021)
- [Learning to Simulate Realistic LiDARs](https://mlsanity.com/paper/2209.10986/) (2022)
- [LEO: Learning Energy-based Models in Factor Graph Optimization](https://mlsanity.com/paper/2108.02274/) (2021)
- [Automatic labelling of urban point clouds using data fusion](https://mlsanity.com/paper/2108.13757/) (2021)
- [SLAM-Supported Self-Training for 6D Object Pose Estimation](https://mlsanity.com/paper/2203.04424/) (2022)

## Built on by (most cited first, 1 of 1)

- [SLACK: Attacking LiDAR-based SLAM with Adversarial Point Injections](https://mlsanity.com/paper/2504.03089/) (2025)

## Abstract (arXiv)

We investigate a new paradigm that uses differentiable SLAM architectures in a self-supervised manner to train end-to-end deep learning models in various LiDAR based applications. To the best of our knowledge there does not exist any work that leverages SLAM as a training signal for deep learning based models. We explore new ways to improve the efficiency, robustness, and adaptability of LiDAR systems with deep learning techniques. We focus on the potential benefits of differentiable SLAM architectures for improving performance of deep learning tasks such as classification, regression as well as SLAM. Our experimental results demonstrate a non-trivial increase in the performance of two deep learning applications - Ground Level Estimation and Dynamic to Static LiDAR Translation, when used with differentiable SLAM architectures. Overall, our findings provide important insights that enhance the performance of LiDAR based navigation systems. We demonstrate that this new paradigm of using SLAM Loss signal while training LiDAR based models can be easily adopted by the community.