mlsanityData to 7 Oct 2026

PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques

arXiv 2304.12410 ↗First posted 24 Apr 2023cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (iii) performance at different downstream tasks, and (iv) how differences in structure and functionality relate to efficiency and task performance. To facilitate such comparisons, this paper presents a reference architecture which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components. Through this process of standardising and isolating differences, a modular view of PEFT techniques emerges, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. We demonstrate how the reference architecture can be applied to understand properties and relative advantages of PEFT techniques, hence to inform selection of techniques for specific tasks, and design choices for new PEFT techniques.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

26 of 38 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 590
14 more
  1. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  3. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  4. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  5. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  6. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  7. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  8. 2023Modular Deep Learningcited by 82
  9. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  10. 2022Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Modelscited by 19
  11. 2022Equivariance Discovery by Learned Parameter-Sharingcited by 18
  12. 2022Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameterscited by 12
  13. 2022Evaluating Parameter Efficient Learning for Generationcited by 3
  14. 2022Attention in Reasoning: Dataset, Analysis, and Modelingcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  2. 2024AlphaZip: Neural Network-Enhanced Lossless Text Compressioncited by 7
  3. 2024Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methodscited by 2
  4. 2024PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuningcited by 0