OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data

arXiv 2511.05028 ↗First posted 7 Nov 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated fine-tuning (FFT) adapts foundation models to decentralized data but remains fragile under heterogeneous client distributions due to local drift, i.e., client-level update divergences that induce systematic bias and amplified variance in the global model. Existing aggregation and personalization methods largely correct drift post hoc, which proves brittle under extreme non-IID conditions. We introduce OvA-LP, a minimalist framework that is, to our knowledge, the first explicitly designed to suppress drift at its source within the PEFT-based FFT paradigm. OvA-LP combines linear probing on a frozen encoder with a one-vs-all head and a simple two-stage procedure, preserving pretrained feature geometry and decoupling logits to prevent the mechanisms that amplify drift. On CIFAR-100 with 100 clients, averaged over shard-1, shard-2, and Bernoulli-Dirichlet partitions, OvA-LP retains 95.9% of its IID accuracy, whereas state-of-the-art FFT baselines retain only 10.1% (PFPT) and 34.5% (FFT-MoE) under the same conditions. OvA-LP further maintains resilience under both symmetric and asymmetric label noise. In addition, precomputing encoder features makes per-round cost nearly independent of encoder size. Together, these results demonstrate that OvA-LP provides a principled and efficient basis for robust FFT under heterogeneity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 7 Nov 2025.

Builds on

19 of 27 references.

References · 19

2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2018Federated Optimization in Heterogeneous Networkscited by 2,971
2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
2016Understanding intermediate layers using linear classifier probescited by 1,279
2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61

7 more

2024Advances and Open Challenges in Federated Foundation Modelscited by 33
2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
2025Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Datacited by 9
2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edgecited by 7
2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
2023FedABC: Targeting Fair Competition in Personalized Federated Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way