mlsanityData to 14 Oct 2026

PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

arXiv 2607.12111 ↗First posted 13 Jul 2026cs.LG cs.AI cs.DCcited by 0

On the map: LLM agents and tool use → Agentic AI for wireless networks

Abstract (arXiv)

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges. Within distributed network environments, Multimodal Large Language Models (MLLMs) serve as cognitive engines for edge devices, yet federated fine-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead. To address these challenges, we propose PFAdapter, a communication-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning. Selective aggregation protocols synchronize only global-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%. Extensive experiments on VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD datasets demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8% across diverse edge intelligence tasks. Consequently, our framework establishes an efficient solution for agentic AI deployment in resource-constrained communication networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agentic AI for wireless networks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 62 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

43 of 62 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
31 more
  1. 2017A Survey on Multi-Task Learningcited by 840
  2. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  3. 2019Federated Learning with Personalization Layerscited by 541
  4. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  7. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  8. 2018A Survey of Unsupervised Deep Domain Adaptationcited by 361
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  10. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  11. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  12. 2018CrisisMMD: Multimodal Twitter Datasets from Natural Disasterscited by 89
  13. 2023Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Modelscited by 59
  14. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  15. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  16. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  17. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  18. 2025A Survey on Federated Fine-tuning of Large Language Modelscited by 24
  19. 2024Networking Systems for Video Anomaly Detection: A Tutorial and Surveycited by 21
  20. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  21. 2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
  22. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  23. 2024FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Datacited by 7
  24. 2023Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNscited by 3
  25. 2024Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networkscited by 3
  26. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  27. 2024Joint Optimization of Resource Allocation and Data Selection for Fast and Cost-Efficient Federated Edge Learningcited by 2
  28. 2026Diffusion-Guided Semantic Consistency for Multimodal Heterogeneitycited by 2
  29. 2026Collaborative Adaptive Curriculum for Progressive Knowledge Distillationcited by 2
  30. 2023Learner Referral for Cost-Effective Federated Learning Over Hierarchical IoT Networkscited by 1
  31. 2025Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiencycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients
  2. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
  3. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
  4. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
  5. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
  6. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning
  7. 2025FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
  8. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning