mlsanityData to 7 Oct 2026

DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model

arXiv 2404.05182 ↗First posted 8 Apr 2024cs.LG cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

To enhance the performance of large language models (LLM) on downstream tasks, one solution is to fine-tune certain LLM parameters and make it better align with the characteristics of the training dataset. This process is commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of LLM, PEFT operations are usually executed in the public environment (e.g., cloud server). This necessitates the sharing of sensitive user data across public environments, thereby raising potential privacy concerns. To tackle these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. Coupled with the proposed Kill and Revive algorithm, the evaluation results demonstrate that DLoRA can significantly reduce the computation and communication workload over the user devices while achieving superior accuracy and privacy protection.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

48 of 63 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
36 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  16. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  17. 2020Adaptive Personalized Federated Learningcited by 353
  18. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  19. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  20. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  21. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  22. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  23. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  24. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  25. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  26. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  27. 2021DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memoriescited by 42
  28. 2020Hermes Attack: Steal DNN Models with Lossless Inference Accuracycited by 41
  29. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  30. 2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 28
  31. 2023RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inferencecited by 12
  32. 2023READ: Recurrent Adaptation of Large Transformerscited by 9
  33. 2023R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Roboticscited by 4
  34. 2023RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamicscited by 4
  35. 2023BP-NTT: Fast and Compact in-SRAM Number Theoretic Transform with Bit-Parallel Modular Multiplicationcited by 3
  36. 2022Inhale: Enabling High-Performance and Energy-Efficient In-SRAM Cryptographic Hash for IoTcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2024When IoT Meet LLMs: Applications and Challengescited by 11
  3. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  4. 2024Efficient Federated Intrusion Detection in 5G ecosystem using optimized BERT-based modelcited by 8
  5. 2025Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryptioncited by 5
  6. 2025UserCentrix: An Agentic Memory-augmented AI Framework for Smart Spacescited by 4
  7. 2024Debiasing Machine Unlearning with Counterfactual Examplescited by 3
  8. 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimizationcited by 0