# On the Federated Learning Framework for Cooperative Perception

arXiv: https://arxiv.org/abs/2404.17147 · HTML: https://mlsanity.com/paper/2404.17147/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 30 of 40)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](https://mlsanity.com/paper/2303.07601/) (2023)
- [Generalized Federated Learning via Sharpness Aware Minimization](https://mlsanity.com/paper/2206.02618/) (2022)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [Collaboration Helps Camera Overtake LiDAR in 3D Detection](https://mlsanity.com/paper/2303.13560/) (2023)
- [HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer](https://mlsanity.com/paper/2304.10628/) (2023)
- [Ground-aware Monocular 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2102.00690/) (2021)
- [LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion](https://mlsanity.com/paper/2307.00724/) (2023)
- [SMURF: Spatial Multi-Representation Fusion for 3D Object Detection with 4D Imaging Radar](https://mlsanity.com/paper/2307.10784/) (2023)
- [Wireless Communications for Collaborative Federated Learning](https://mlsanity.com/paper/2006.02499/) (2020)
- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [Among Us: Adversarially Robust Collaborative Perception by Consensus](https://mlsanity.com/paper/2303.09495/) (2023)
- [FedTP: Federated Learning by Transformer Personalization](https://mlsanity.com/paper/2211.01572/) (2022)
- [RaLiBEV: Radar and LiDAR BEV Fusion Learning for Anchor Box Free Object Detection Systems](https://mlsanity.com/paper/2211.06108/) (2022)
- [Deep Instance Segmentation with Automotive Radar Detection Points](https://mlsanity.com/paper/2110.01775/) (2021)
- [ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking](https://mlsanity.com/paper/2211.03919/) (2022)
- [SCENE: Reasoning about Traffic Scenes using Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2301.03512/) (2023)
- [A Survey of Federated Learning for Connected and Automated Vehicles](https://mlsanity.com/paper/2303.10677/) (2023)
- [GNN-PMB: A Simple but Effective Online 3D Multi-Object Tracker without Bells and Whistles](https://mlsanity.com/paper/2206.10255/) (2022)
- [Infrastructure-Based Object Detection and Tracking for Cooperative Driving Automation: A Survey](https://mlsanity.com/paper/2201.11871/) (2022)

## Built on by (most cited first, 2 of 2)

- [Enabling Federated Object Detection for Connected Autonomous Vehicles: A Deployment-Oriented Evaluation](https://mlsanity.com/paper/2509.01868/) (2025)
- [Sarus: Privacy-Preserving Multi-Vendor Perception Fusion via Homomorphic Encryption](https://mlsanity.com/paper/2607.19146/) (2026)

## Abstract (arXiv)

Cooperative perception is essential to enhance the efficiency and safety of future transportation systems, requiring extensive data sharing among vehicles on the road, which raises significant privacy concerns. Federated learning offers a promising solution by enabling data privacy-preserving collaborative enhancements in perception, decision-making, and planning among connected and autonomous vehicles (CAVs). However, federated learning is impeded by significant challenges arising from data heterogeneity across diverse clients, potentially diminishing model accuracy and prolonging convergence periods. This study introduces a specialized federated learning framework for CP, termed the federated dynamic weighted aggregation (FedDWA) algorithm, facilitated by dynamic adjusting loss (DALoss) function. This framework employs dynamic client weighting to direct model convergence and integrates a novel loss function that utilizes Kullback-Leibler divergence (KLD) to counteract the detrimental effects of non-independently and identically distributed (Non-IID) and unbalanced data. Utilizing the BEV transformer as the primary model, our rigorous testing on the OpenV2V dataset, augmented with FedBEVT data, demonstrates significant improvements in the average intersection over union (IoU). These results highlight the substantial potential of our federated learning framework to address data heterogeneity challenges in CP, thereby enhancing the accuracy of environmental perception models and facilitating more robust and efficient collaborative learning solutions in the transportation sector.