mlsanityData to 14 Oct 2026

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

arXiv 2607.29071 ↗First posted 31 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 73 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

57 of 73 references.

References · 57

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
38 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2023BloombergGPT: A Large Language Model for Financecited by 939
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  5. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  10. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  11. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  12. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  13. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  14. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  15. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  16. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  17. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  18. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  19. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  20. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  21. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  22. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  23. 2021Decentralized Federated Averagingcited by 114
  24. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  25. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  26. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  27. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  28. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  29. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 66
  30. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 63
  31. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  32. 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
  33. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  34. 2025Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectivescited by 48
  35. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  36. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  37. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
  38. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation
  2. 2024Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes
  3. 2025OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data
  4. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
  5. 2023FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications
  6. 2026Tackling Resource-Constrained and Data-Heterogeneity in Federated Learning with Double-Weight Sparse Pack
  7. 2024Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation
  8. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning