mlsanityData to 14 Oct 2026

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

arXiv 2304.01933 ↗First posted 4 Apr 2023cs.CLcited by 324

Not on the map. Why?

Abstract (arXiv)

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.

Key facts. Cited by 324 papers in the map (data through 14 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

39 of 57 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
27 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  13. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  14. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  15. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  17. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  18. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  19. 2016Solving General Arithmetic Word Problemscited by 360
  20. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  21. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  22. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  23. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  24. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  25. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  26. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  27. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 31; 2024: 110; 2025: 120; 2026: 63.

324 papers.

Sort:

By area:

  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  7. 2023Challenges and Applications of Large Language Modelscited by 254
  8. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  9. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  10. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  12. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
38 more
  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  2. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  3. 2024Large Language Models Meet NLP: A Surveycited by 103
  4. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  5. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  6. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  7. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  8. 2025The Curse of Depth in Large Language Modelscited by 62
  9. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  10. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  11. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  12. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  13. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  14. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  15. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  16. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  17. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  18. 2024MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learningcited by 44
  19. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  20. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
  21. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  22. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  23. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  24. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  25. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  26. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 35
  27. 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 32
  28. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  29. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  30. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  31. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  32. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  33. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  34. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  35. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  36. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  37. 2024Octopus: On-device language model for function calling of software APIscited by 27
  38. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 27

50 most cited of 324 All 324 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  2. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models
  3. 2024Exploring Advanced Large Language Models with LLMsuite
  4. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  5. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter
  6. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  7. 2024Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning
  8. 2025Slimming Down LLMs Without Losing Their Minds