# LAPTNet: LiDAR-Aided Perspective Transform Network

arXiv: https://arxiv.org/abs/2211.14445 · HTML: https://mlsanity.com/paper/2211.14445/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 14 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [FISHING Net: Future Inference of Semantic Heatmaps In Grids](https://mlsanity.com/paper/2006.09917/) (2020)
- [Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles](https://mlsanity.com/paper/1904.08494/) (2019)
- [PillarSegNet: Pillar-based Semantic Grid Map Estimation using Sparse LiDAR Data](https://mlsanity.com/paper/2105.04169/) (2021)
- [Short-Term Prediction and Multi-Camera Fusion on Semantic Grids](https://mlsanity.com/paper/1903.08960/) (2019)

## Abstract (arXiv)

Semantic grids are a useful representation of the environment around a robot. They can be used in autonomous vehicles to concisely represent the scene around the car, capturing vital information for downstream tasks like navigation or collision assessment. Information from different sensors can be used to generate these grids. Some methods rely only on RGB images, whereas others choose to incorporate information from other sensors, such as radar or LiDAR. In this paper, we present an architecture that fuses LiDAR and camera information to generate semantic grids. By using the 3D information from a LiDAR point cloud, the LiDAR-Aided Perspective Transform Network (LAPTNet) is able to associate features in the camera plane to the bird's eye view without having to predict any depth information about the scene. Compared to state-of-theart camera-only methods, LAPTNet achieves an improvement of up to 8.8 points (or 38.13%) over state-of-art competing approaches for the classes proposed in the NuScenes dataset validation split.