# pFedLVM: A Large Vision Model (LVM)-Driven and Latent Feature-Based Personalized Federated Learning Framework in Autonomous Driving

arXiv: https://arxiv.org/abs/2405.04146 · HTML: https://mlsanity.com/paper/2405.04146/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2024.

## Built on by (most cited first, 16 of 16)

- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving](https://mlsanity.com/paper/2409.19560/) (2024)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2407.13111/) (2024)
- [XLM for Autonomous Driving Systems: A Comprehensive Review](https://mlsanity.com/paper/2409.10484/) (2024)
- [FedEMA: Federated Exponential Moving Averaging with Negative Entropy Regularizer in Autonomous Driving](https://mlsanity.com/paper/2505.00318/) (2025)
- [Label Anything: An Interpretable, High-Fidelity and Prompt-Free Annotator](https://mlsanity.com/paper/2502.02972/) (2025)
- [A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environments](https://mlsanity.com/paper/2502.10419/) (2025)
- [FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization](https://mlsanity.com/paper/2505.09385/) (2025)
- [FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios](https://mlsanity.com/paper/2510.11274/) (2025)
- [FedDSR: Federated Deep Supervision and Regularization Towards Autonomous Driving](https://mlsanity.com/paper/2512.06676/) (2025)
- [Authorize-on-Demand: Dynamic Authorization with Legality-Aware Intellectual Property Protection for VLMs](https://mlsanity.com/paper/2603.04896/) (2026)
- [OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization](https://mlsanity.com/paper/2605.20276/) (2026)
- [FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models](https://mlsanity.com/paper/2605.28347/) (2026)
- [Federated Foundation Models over Vehicular Networks](https://mlsanity.com/paper/2606.06786/) (2026)
- [AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligence](https://mlsanity.com/paper/2607.16562/) (2026)

## Abstract (arXiv)

Deep learning-based Autonomous Driving (AD) models often exhibit poor generalization due to data heterogeneity in an ever domain-shifting environment. While Federated Learning (FL) could improve the generalization of an AD model (known as FedAD system), conventional models often struggle with under-fitting as the amount of accumulated training data progressively increases. To address this issue, instead of conventional small models, employing Large Vision Models (LVMs) in FedAD is a viable option for better learning of representations from a vast volume of data. However, implementing LVMs in FedAD introduces three challenges: (I) the extremely high communication overheads associated with transmitting LVMs between participating vehicles and a central server; (II) lack of computing resource to deploy LVMs on each vehicle; (III) the performance drop due to LVM focusing on shared features but overlooking local vehicle characteristics. To overcome these challenges, we propose pFedLVM, a LVM-Driven, Latent Feature-Based Personalized Federated Learning framework. In this approach, the LVM is deployed only on central server, which effectively alleviates the computational burden on individual vehicles. Furthermore, the exchange between central server and vehicles are the learned features rather than the LVM parameters, which significantly reduces communication overhead. In addition, we utilize both shared features from all participating vehicles and individual characteristics from each vehicle to establish a personalized learning mechanism. This enables each vehicle's model to learn features from others while preserving its personalized characteristics, thereby outperforming globally shared models trained in general FL. Extensive experiments demonstrate that pFedLVM outperforms the existing state-of-the-art approaches.