# Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving

arXiv: https://arxiv.org/abs/2109.11615 · HTML: https://mlsanity.com/paper/2109.11615/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction](https://mlsanity.com/paper/2008.07519/) (2020)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds](https://mlsanity.com/paper/1909.06459/) (2019)
- [Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds](https://mlsanity.com/paper/1905.05265/) (2019)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [CIA-SSD: Confident IoU-Aware Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2012.03015/) (2020)
- [Sparse 3D convolutional neural networks](https://mlsanity.com/paper/1505.02890/) (2015)
- [Demonstrations of Cooperative Perception: Safety and Robustness in Connected and Automated Vehicle Operations](https://mlsanity.com/paper/2011.08581/) (2020)
- [Feature Sharing and Integration for Cooperative Cognition and Perception with Volumetric Sensors](https://mlsanity.com/paper/2011.08317/) (2020)

## Built on by (most cited first, 30 of 45)

- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022)
- [Robust Collaborative 3D Object Detection in Presence of Pose Errors](https://mlsanity.com/paper/2211.07214/) (2022)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [Latency-Aware Collaborative Perception](https://mlsanity.com/paper/2207.08560/) (2022)
- [Bridging the Domain Gap for Multi-Agent Perception](https://mlsanity.com/paper/2210.08451/) (2022)
- [CORE: Cooperative Reconstruction for Multi-Agent Perception](https://mlsanity.com/paper/2307.11514/) (2023)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception](https://mlsanity.com/paper/2403.10145/) (2024)
- [V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception](https://mlsanity.com/paper/2209.13679/) (2022)
- [Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving](https://mlsanity.com/paper/2304.11821/) (2023)
- [Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?](https://mlsanity.com/paper/2306.09281/) (2023)
- [A Survey and Framework of Cooperative Perception: From Heterogeneous Singleton to Hierarchical Cooperation](https://mlsanity.com/paper/2208.10590/) (2022)
- [BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities](https://mlsanity.com/paper/2310.14702/) (2023)
- [DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception](https://mlsanity.com/paper/2310.08117/) (2023)
- [Vehicle-to-Everything Cooperative Perception for Autonomous Driving](https://mlsanity.com/paper/2310.03525/) (2023)
- [SparseAlign: A Fully Sparse Framework for Cooperative Object Detection](https://mlsanity.com/paper/2503.12982/) (2025)
- [A Cooperative Perception System Robust to Localization Errors](https://mlsanity.com/paper/2210.06289/) (2022)
- [A Survey on Intermediate Fusion Methods for Collaborative Perception Categorized by Real World Challenges](https://mlsanity.com/paper/2404.16139/) (2024)
- [On Data Fabrication in Collaborative Vehicular Perception: Attacks and Countermeasures](https://mlsanity.com/paper/2309.12955/) (2023)
- [Collaborative Perception for Autonomous Driving: Current Status and Future Trend](https://mlsanity.com/paper/2208.10371/) (2022)
- [Generating Evidential BEV Maps in Continuous Driving Space](https://mlsanity.com/paper/2302.02928/) (2023)
- [Collaborative Perception Datasets for Autonomous Driving: A Review](https://mlsanity.com/paper/2504.12696/) (2025)
- [Collective PV-RCNN: A Novel Fusion Technique using Collective Detections for Enhanced Local LiDAR-Based Perception](https://mlsanity.com/paper/2309.05380/) (2023)
- [Systematic Literature Review on Vehicular Collaborative Perception -- A Computer Vision Perspective](https://mlsanity.com/paper/2504.04631/) (2025)
- [Which2comm: An Efficient Collaborative Perception Framework for 3D Object Detection](https://mlsanity.com/paper/2503.17175/) (2025)
- [V2X-DGPE: Addressing Domain Gaps and Pose Errors for Robust Collaborative 3D Object Detection](https://mlsanity.com/paper/2501.02363/) (2025)
- [CoBEVMoE: Heterogeneity-aware Feature Fusion with Dynamic Mixture-of-Experts for Collaborative Perception](https://mlsanity.com/paper/2509.17107/) (2025)
- [An End-to-End Framework of Road User Detection, Tracking, and Prediction from Monocular Images](https://mlsanity.com/paper/2308.05026/) (2023)

## Abstract (arXiv)

Sharing collective perception messages (CPM) between vehicles is investigated to decrease occlusions so as to improve the perception accuracy and safety of autonomous driving. However, highly accurate data sharing and low communication overhead is a big challenge for collective perception, especially when real-time communication is required among connected and automated vehicles. In this paper, we propose an efficient and effective keypoints-based deep feature fusion framework built on the 3D object detector PV-RCNN, called Fusion PV-RCNN (FPV-RCNN for short), for collective perception. We introduce a high-performance bounding box proposal matching module and a keypoints selection strategy to compress the CPM size and solve the multi-vehicle data fusion problem. Besides, we also propose an effective localization error correction module based on the maximum consensus principle to increase the robustness of the data fusion. Compared to a bird's-eye view (BEV) keypoints feature fusion, FPV-RCNN achieves improved detection accuracy by about 9% at a high evaluation criterion (IoU 0.7) on the synthetic dataset COMAP dedicated to collective perception. In addition, its performance is comparable to two raw data fusion baselines that have no data loss in sharing. Moreover, our method also significantly decreases the CPM size to less than 0.3 KB, and is thus about 50 times smaller than the BEV feature map sharing used in previous works. Even with further decreased CPM feature channels, i.e., from 128 to 32, the detection performance does not show apparent drops. The code of our method is available at https://github.com/YuanYunshuang/FPV_RCNN.