mlsanityData to 14 Oct 2026

Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models

arXiv 2305.16597 ↗First posted 26 May 2023cs.CL cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

38 of 52 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
26 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  9. 2017Learning multiple visual domains with residual adapterscited by 557
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  12. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  13. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  14. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  15. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  16. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  17. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  19. 2019The Evolved Transformercited by 180
  20. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  21. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  22. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  23. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  24. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  25. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  26. 2021Differentiable Subset Pruning of Transformer Headscited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 7; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  5. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  6. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 14
  7. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  8. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 11
  9. 2023Exploring the Impact of Model Scaling on Parameter-Efficient Tuningcited by 8
  10. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  11. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  12. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
13 more
  1. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  2. 2024Personalized Federated Instruction Tuning via Neural Architecture Searchcited by 3
  3. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  4. 2024Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Predictioncited by 3
  5. 2024Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Modelscited by 2
  6. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  7. 2026Bilevel Optimization for Neural Architecture Searchcited by 2
  8. 2024QuAILoRA: Quantization-Aware Initialization for LoRAcited by 0
  9. 2025PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMscited by 0
  10. 2025GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Modelscited by 0
  11. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0
  12. 2026Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuningcited by 0
  13. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0