# Sensor Fusion by Spatial Encoding for Autonomous Driving

arXiv: https://arxiv.org/abs/2308.10707 · HTML: https://mlsanity.com/paper/2308.10707/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection](https://mlsanity.com/paper/2009.00784/) (2020)
- [Multi-task Learning with Attention for End-to-end Autonomous Driving](https://mlsanity.com/paper/2104.10753/) (2021)
- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [RelMobNet: End-to-end relative camera pose estimation using a robust two-stage training](https://mlsanity.com/paper/2202.12838/) (2022)
- [Learning Dense Features for Point Cloud Registration Using a Graph Attention Network](https://mlsanity.com/paper/2206.06731/) (2022)
- [A Lightweight Domain Adaptive Absolute Pose Regressor Using Barlow Twins Objective](https://mlsanity.com/paper/2211.10963/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directions](https://mlsanity.com/paper/2403.07542/) (2024)
- [PoseViNet: Distracted Driver Action Recognition Framework Using Multi-View Pose Estimation and Vision Transformer](https://mlsanity.com/paper/2312.14577/) (2023)
- [Enhanced Transformer Architecture for Natural Language Processing](https://mlsanity.com/paper/2310.10930/) (2023)

## Abstract (arXiv)

Sensor fusion is critical to perception systems for task domains such as autonomous driving and robotics. Recently, the Transformer integrated with CNN has demonstrated high performance in sensor fusion for various perception tasks. In this work, we introduce a method for fusing data from camera and LiDAR. By employing Transformer modules at multiple resolutions, proposed method effectively combines local and global contextual relationships. The performance of the proposed method is validated by extensive experiments with two adversarial benchmarks with lengthy routes and high-density traffics. The proposed method outperforms previous approaches with the most challenging benchmarks, achieving significantly higher driving and infraction scores. Compared with TransFuser, it achieves 8% and 19% improvement in driving scores for the Longest6 and Town05 Long benchmarks, respectively.