# MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views

arXiv: https://arxiv.org/abs/2006.05518 · HTML: https://mlsanity.com/paper/2006.05518/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention](https://mlsanity.com/paper/2203.09704/) (2022)
- [GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2207.02605/) (2022)
- [LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation](https://mlsanity.com/paper/2108.07511/) (2021)
- [MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data](https://mlsanity.com/paper/2104.10772/) (2021)
- [NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving](https://mlsanity.com/paper/2209.14499/) (2022)
- [RestoreX-AI: A Contrastive Approach towards Guiding Image Restoration via Explainable AI Systems](https://mlsanity.com/paper/2204.01719/) (2022)
- [M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving](https://mlsanity.com/paper/2403.12552/) (2024)
- [In Rain or Shine: Understanding and Overcoming Dataset Bias for Improving Robustness Against Weather Corruptions for Autonomous Vehicles](https://mlsanity.com/paper/2204.01062/) (2022)
- [A Review of Mobile Mapping Systems: From Sensors to Applications](https://mlsanity.com/paper/2205.15865/) (2022)
- [Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2304.14800/) (2023)
- [What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2309.07808/) (2023)
- [Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving](https://mlsanity.com/paper/2210.06758/) (2022)
- [Beyond Traditional Single Object Tracking: A Survey](https://mlsanity.com/paper/2405.10439/) (2024)
- [Multimodal Point Cloud Semantic Segmentation With Virtual Point Enhancement](https://mlsanity.com/paper/2504.01449/) (2025)
- [From Spherical to Gaussian: A Comparative Analysis of Point Cloud Cropping Strategies in Large-Scale 3D Environments](https://mlsanity.com/paper/2605.02098/) (2026)
- [CORAL: Curriculum-Optimized Reward Adaptation for LiDAR-Based Goal-Directed Urban Driving](https://mlsanity.com/paper/2608.14332/) (2026)

## Abstract (arXiv)

Autonomous driving requires the inference of actionable information such as detecting and classifying objects, and determining the drivable space. To this end, we present Multi-View LidarNet (MVLidarNet), a two-stage deep neural network for multi-class object detection and drivable space segmentation using multiple views of a single LiDAR point cloud. The first stage processes the point cloud projected onto a perspective view in order to semantically segment the scene. The second stage then processes the point cloud (along with semantic labels from the first stage) projected onto a bird's eye view, to detect and classify objects. Both stages use an encoder-decoder architecture. We show that our multi-view, multi-stage, multi-class approach is able to detect and classify objects while simultaneously determining the drivable space using a single LiDAR scan as input, in challenging scenes with more than one hundred vehicles and pedestrians at a time. The system operates efficiently at 150 fps on an embedded GPU designed for a self-driving car, including a postprocessing step to maintain identities over time. We show results on both KITTI and a much larger internal dataset, thus demonstrating the method's ability to scale by an order of magnitude.