mlsanityData to 7 Oct 2026

Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning

arXiv 2004.03829 ↗First posted 8 Apr 2020cs.CLcited by 139

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pre-trained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 18; 2023: 32; 2024: 54; 2025: 23; 2026: 5.

139 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  5. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  7. 2023Personalize Segment Anything Model with One Shotcited by 250
  8. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  9. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  10. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  11. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  12. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
38 more
  1. 2021Sustainable Modular Debiasing of Language Modelscited by 98
  2. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  3. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  4. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  5. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 67
  6. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  7. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  8. 2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
  9. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  10. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 51
  11. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  12. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  13. 2024Training of Physical Neural Networkscited by 47
  14. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  15. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  16. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  17. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  18. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  19. 2023Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphscited by 35
  20. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  21. 2023Visual Tuningcited by 34
  22. 2021Dimensions of Commonsense Knowledgecited by 29
  23. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  24. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 28
  25. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  26. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  27. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  28. 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24
  29. 2022Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generationcited by 22
  30. 2023LION: Implicit Vision Prompt Tuningcited by 22
  31. 2022Plug-and-Play Adaptation for Continuously-updated QAcited by 21
  32. 2023Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shiftscited by 21
  33. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  34. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  35. 2024Low-Rank Adaptation of Time Series Foundational Models for Out-of-Domain Modality Forecastingcited by 18
  36. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  37. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 16
  38. 2023TADA: Task-Agnostic Dialect Adapters for Englishcited by 15

50 most cited of 139 All 139 (JSON)