# Collaborative 3D Object Detection for Automatic Vehicle Systems via Learnable Communications

arXiv: https://arxiv.org/abs/2205.11849 · HTML: https://mlsanity.com/paper/2205.11849/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds](https://mlsanity.com/paper/1909.06459/) (2019)
- [Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds](https://mlsanity.com/paper/1905.05265/) (2019)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [Who2com: Collaborative Perception via Learnable Handshake Communication](https://mlsanity.com/paper/2003.09575/) (2020)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors](https://mlsanity.com/paper/1912.12147/) (2019)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [CoFF: Cooperative Spatial Feature Fusion for 3D Object Detection on Autonomous Vehicles](https://mlsanity.com/paper/2009.11975/) (2020)

## Built on by (most cited first, 4 of 4)

- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [Adaptive Communications in Collaborative Perception with Domain Alignment for Autonomous Driving](https://mlsanity.com/paper/2310.00013/) (2023)
- [Vehicle-to-Everything Cooperative Perception for Autonomous Driving](https://mlsanity.com/paper/2310.03525/) (2023)
- [EI-Drive: A Platform for Cooperative Perception with Realistic Communication Models](https://mlsanity.com/paper/2412.09782/) (2024)

## Abstract (arXiv)

Accurate detection of objects in 3D point clouds is a key problem in autonomous driving systems. Collaborative perception can incorporate information from spatially diverse sensors and provide significant benefits for improving the perception accuracy of autonomous driving systems. In this work, we consider that the autonomous vehicle uses local point cloud data and combines information from neighboring infrastructures through wireless links for cooperative 3D object detection. However, information sharing among vehicle and infrastructures in predefined communication schemes may result in communication congestion and/or bring limited performance improvement. To this end, we propose a novel collaborative 3D object detection framework that consists of three components: feature learning networks that map point clouds into feature maps; an efficient communication block that propagates compact and fine-grained query feature maps from vehicle to support infrastructures and optimizes attention weights between query and key to refine support feature maps; a region proposal network that fuses local feature maps and weighted support feature maps for 3D object detection. We evaluate the performance of the proposed framework using a synthetic cooperative dataset created in two complex driving scenarios: a roundabout and a T-junction. Experiment results and bandwidth usage analysis demonstrate that our approach can save communication and computation costs and significantly improve detection performance under different detection difficulties in all scenarios.