# CoSense3D: an Agent-based Efficient Learning Framework for Collective Perception

arXiv: https://arxiv.org/abs/2404.18617 · HTML: https://mlsanity.com/paper/2404.18617/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds](https://mlsanity.com/paper/1909.06459/) (2019)
- [CIA-SSD: Confident IoU-Aware Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2012.03015/) (2020)
- [Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving](https://mlsanity.com/paper/2109.11615/) (2021)
- [Generating Evidential BEV Maps in Continuous Driving Space](https://mlsanity.com/paper/2302.02928/) (2023)

## Abstract (arXiv)

Collective Perception has attracted significant attention in recent years due to its advantage for mitigating occlusion and expanding the field-of-view, thereby enhancing reliability, efficiency, and, most crucially, decision-making safety. However, developing collective perception models is highly resource demanding due to extensive requirements of processing input data for many agents, usually dozens of images and point clouds for a single frame. This not only slows down the model development process for collective perception but also impedes the utilization of larger models. In this paper, we propose an agent-based training framework that handles the deep learning modules and agent data separately to have a cleaner data flow structure. This framework not only provides an API for flexibly prototyping the data processing pipeline and defining the gradient calculation for each agent, but also provides the user interface for interactive training, testing and data visualization. Training experiment results of four collective object detection models on the prominent collective perception benchmark OPV2V show that the agent-based training can significantly reduce the GPU memory consumption and training time while retaining inference performance. The framework and model implementations are available at urlhttps://github.com/YuanYunshuang/CoSense3D