mlsanityData to 14 Oct 2026

DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

arXiv 2505.21382 ↗First posted 27 May 2025cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has emerged as one of the most effective, computationally tractable fine-tuning approaches for training Vision-Language Models (VLMs) and Large Language Models (LLMs). LoRA accomplishes this by freezing the pre-trained model weights and injecting trainable low-rank matrices, allowing for efficient learning of these foundation models even on edge devices. However, LoRA in decentralized settings still remains under explored, particularly for the theoretical underpinnings due to the lack of smoothness guarantee and model consensus interference (defined formally below). This work improves the convergence rate of decentralized LoRA (DLoRA) to match the rate of decentralized SGD by ensuring gradient smoothness. We also introduce DeCAF, a novel algorithm integrating DLoRA with truncated singular value decomposition (TSVD)-based matrix factorization to resolve consensus interference. Theoretical analysis shows TSVD's approximation error is bounded and consensus differences between DLoRA and DeCAF vanish as rank increases, yielding DeCAF's matching convergence rate. Extensive experiments across vision/language tasks demonstrate our algorithms outperform local training and rivals federated learning under both IID and non-IID data distributions.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 71 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

51 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2019Towards Federated Learning at Scale: System Designcited by 832
38 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  3. 2017Learning multiple visual domains with residual adapterscited by 557
  4. 2018Sparsified SGD with Memorycited by 376
  5. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  6. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  7. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  8. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  9. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  11. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  12. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  13. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  14. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  15. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  16. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  17. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 138
  18. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  19. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  20. 2023Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?cited by 106
  21. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  22. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  23. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  24. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  25. 2020Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)cited by 54
  26. 2016On Nonconvex Decentralized Gradient Descentcited by 42
  27. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  28. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 33
  29. 2024Decentralized Federated Learning: A Survey on Security and Privacycited by 25
  30. 2017Balancing Communication and Computation in Distributed Optimizationcited by 23
  31. 2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Modelscited by 15
  32. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  33. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 13
  34. 2024MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Modelscited by 10
  35. 2024Personalized Collaborative Fine-Tuning for On-Device Large Language Modelscited by 9
  36. 2023FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Modelscited by 4
  37. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  38. 2024DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  2. 2026DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  2. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  3. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
  4. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
  5. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  6. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  7. 2025Uni-LoRA: One Vector is All You Need
  8. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning