# Federated Deep Learning Meets Autonomous Vehicle Perception: Design and Verification

arXiv: https://arxiv.org/abs/2206.01748 · HTML: https://mlsanity.com/paper/2206.01748/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 3 Jun 2022.

## Builds on (most cited first, 11 of 11)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Learning Distilled Collaboration Graph for Multi-Agent Perception](https://mlsanity.com/paper/2111.00643/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors](https://mlsanity.com/paper/1912.12147/) (2019)
- [FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring](https://mlsanity.com/paper/2012.07450/) (2020)
- [Opportunities of Federated Learning in Connected, Cooperative and Automated Industrial Systems](https://mlsanity.com/paper/2101.03367/) (2021)
- [Federated Imitation Learning: A Novel Framework for Cloud Robotic Systems with Heterogeneous Sensor Data](https://mlsanity.com/paper/1912.12204/) (2019)
- [Mobile Edge Intelligence and Computing for the Internet of Vehicles](https://mlsanity.com/paper/1906.00400/) (2019)
- [Finding Critical Scenarios for Automated Driving Systems: A Systematic Literature Review](https://mlsanity.com/paper/2110.08664/) (2021)
- [On-edge Multi-task Transfer Learning: Model and Practice with Data-driven Task Allocation](https://mlsanity.com/paper/2107.02466/) (2021)

## Built on by (most cited first, 14 of 14)

- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [Mobility-Aware Cooperative Caching in Vehicular Edge Computing Based on Asynchronous Federated and Deep Reinforcement Learning](https://mlsanity.com/paper/2208.01219/) (2022)
- [A Survey of Federated Learning for Connected and Automated Vehicles](https://mlsanity.com/paper/2303.10677/) (2023)
- [Vehicle-to-Everything Cooperative Perception for Autonomous Driving](https://mlsanity.com/paper/2310.03525/) (2023)
- [FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems](https://mlsanity.com/paper/2304.01534/) (2023)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [Communication Resources Constrained Hierarchical Federated Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2306.16169/) (2023)
- [FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units](https://mlsanity.com/paper/2401.12862/) (2024)
- [FedPylot: Navigating Federated Learning for Real-Time Object Detection in Internet of Vehicles](https://mlsanity.com/paper/2406.03611/) (2024)
- [iCOIL: Scenario Aware Autonomous Parking Via Integrated Constrained Optimization and Imitation Learning](https://mlsanity.com/paper/2305.13663/) (2023)
- [Personalized Federated Learning for Cross-view Geo-localization](https://mlsanity.com/paper/2411.04692/) (2024)
- [Enabling Federated Object Detection for Connected Autonomous Vehicles: A Deployment-Oriented Evaluation](https://mlsanity.com/paper/2509.01868/) (2025)
- [Hierarchical Multi-Task Federated Learning in VANETs](https://mlsanity.com/paper/2608.08111/) (2026)
- [Encoder-Sharing Hierarchical Federated Multi-Task Learning for VANETs](https://mlsanity.com/paper/2609.36157/) (2026)

## Abstract (arXiv)

Realizing human-like perception is a challenge in open driving scenarios due to corner cases and visual occlusions. To gather knowledge of rare and occluded instances, federated learning assisted connected autonomous vehicle (FLCAV) has been proposed, which leverages vehicular networks to establish federated deep neural networks (DNNs) from distributed data captured by vehicles and road sensors. Without the need of data aggregation, FLCAV preserves privacy while reducing communication costs compared with conventional centralized learning. However, it is challenging to determine the network resources and road sensor placements for multi-stage training with multi-modal datasets in multi-variant scenarios. This article presents networking and training frameworks for FLCAV perception. Multi-layer graph resource allocation and vehicle-road contrastive sensor placement are proposed to address the network management and sensor deployment problems, respectively. We also develop CarlaFLCAV, a software platform that implements the above system and methods. Experimental results confirm the superiority of the proposed techniques compared with various benchmarks.