mlsanityData to 7 Oct 2026

Parameter-Efficient Transfer Learning for NLP

arXiv 1902.00751 ↗First posted 2 Feb 2019cs.LG cs.CL stat.MLcited by 4,195

Not on the map. Why?

Cited by 210 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

Key facts. Cited by 4,195 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 165; 2022: 413; 2023: 814; 2024: 1136; 2025: 900; 2026: 767.

4,195 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2022Visual Prompt Tuningcited by 1,674
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
38 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  13. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  14. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  15. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  16. 2024A Survey on Large Language Models for Code Generationcited by 646
  17. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  18. 2023A Comprehensive Overview of Large Language Modelscited by 639
  19. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  20. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  21. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  22. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  23. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  24. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  25. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  26. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  27. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  28. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  29. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  30. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  31. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  32. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  33. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  34. 2023Segment Anything in High Qualitycited by 373
  35. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  36. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  37. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  38. 2021Random Feature Attentioncited by 350

50 most cited of 4,195 All 4,195 (JSON)