# Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

arXiv: https://arxiv.org/abs/2008.11901 · HTML: https://mlsanity.com/paper/2008.11901/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2020.

## Built on by (most cited first, 30 of 41)

- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation](https://mlsanity.com/paper/2211.15656/) (2022)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data](https://mlsanity.com/paper/2104.10772/) (2021)
- [Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs](https://mlsanity.com/paper/2404.04869/) (2024)
- [Deep Transfer Learning for Intelligent Vehicle Perception: a Survey](https://mlsanity.com/paper/2306.15110/) (2023)
- [VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer](https://mlsanity.com/paper/2311.06623/) (2023)
- [Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking](https://mlsanity.com/paper/2206.03666/) (2022)
- [MGMapNet: Multi-Granularity Representation Learning for End-to-End Vectorized HD Map Construction](https://mlsanity.com/paper/2410.07733/) (2024)
- [Constraining Depth Map Geometry for Multi-View Stereo: A Dual-Depth Approach with Saddle-shaped Depth Cells](https://mlsanity.com/paper/2307.09160/) (2023)
- [MoST: Multi-modality Scene Tokenization for Motion Prediction](https://mlsanity.com/paper/2404.19531/) (2024)
- [M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving](https://mlsanity.com/paper/2403.12552/) (2024)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models](https://mlsanity.com/paper/2405.14271/) (2024)
- [Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles](https://mlsanity.com/paper/2408.11196/) (2024)
- [RopeBEV: A Multi-Camera Roadside Perception Network in Bird's-Eye-View](https://mlsanity.com/paper/2409.11706/) (2024)
- [Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2502.07631/) (2025)
- [Spatial Retrieval Augmented Autonomous Driving](https://mlsanity.com/paper/2512.06865/) (2025)
- [Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models](https://mlsanity.com/paper/2101.03279/) (2021)
- [MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving](https://mlsanity.com/paper/2405.07573/) (2024)
- [Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?](https://mlsanity.com/paper/2412.08973/) (2024)
- [What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2309.07808/) (2023)
- [SpotNet: An Image Centric, Lidar Anchored Approach To Long Range Perception](https://mlsanity.com/paper/2405.15843/) (2024)
- [Robust sensor fusion against on-vehicle sensor staleness](https://mlsanity.com/paper/2506.05780/) (2025)
- [Ego-centric Learning of Communicative World Models for Autonomous Driving](https://mlsanity.com/paper/2506.08149/) (2025)

## Abstract (arXiv)

We present an end-to-end method for object detection and trajectory prediction utilizing multi-view representations of LiDAR returns and camera images. In this work, we recognize the strengths and weaknesses of different view representations, and we propose an efficient and generic fusing method that aggregates benefits from all views. Our model builds on a state-of-the-art Bird's-Eye View (BEV) network that fuses voxelized features from a sequence of historical LiDAR data as well as rasterized high-definition map to perform detection and prediction tasks. We extend this model with additional LiDAR Range-View (RV) features that use the raw LiDAR information in its native, non-quantized representation. The RV feature map is projected into BEV and fused with the BEV features computed from LiDAR and high-definition map. The fused features are then further processed to output the final detections and trajectories, within a single end-to-end trainable network. In addition, the RV fusion of LiDAR and camera is performed in a straightforward and computationally efficient manner using this framework. The proposed multi-view fusion approach improves the state-of-the-art on proprietary large-scale real-world data collected by a fleet of self-driving vehicles, as well as on the public nuScenes data set with minimal increases on the computational cost.