mlsanityData to 14 Oct 2026

Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper

arXiv 2311.13126 ↗First posted 22 Nov 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper delves into the pressing need in Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs). While LLMs possess remarkable capabilities, their extensive parameter requirements and associated computational demands hinder their practicality and scalability for real-world applications. Our position paper highlights current states and the necessity of further studying into the topic, and recognizes significant challenges and open issues that must be addressed to fully harness the powerful abilities of LLMs. These challenges encompass novel efficient PEFT architectures, PEFT for different learning settings, PEFT combined with model compression techniques, and the exploration of PEFT for multi-modal LLMs. By presenting this position paper, we aim to stimulate further research and foster discussions surrounding more efficient and accessible PEFT for LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

40 of 54 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
28 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2022Galactica: A Large Language Model for Sciencecited by 744
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  11. 2021GPT Understands, Toocited by 716
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  13. 2018CoQA: A Conversational Question Answering Challengecited by 683
  14. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  15. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  16. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  17. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  18. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  19. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  20. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  21. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  22. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  23. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  24. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 23
  25. 2022Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuningcited by 19
  26. 2023Prefix Propagation: Parameter-Efficient Tuning for Long Sequencescited by 15
  27. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
  28. 2022Evaluating Parameter Efficient Learning for Generationcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  2. 2024Tuning a SAM-Based Model with Multi-Cognitive Visual Adapter to Remote Sensing Instance Segmentationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models
  2. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  3. 2024Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models
  4. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
  5. 2024HFT: Half Fine-Tuning for Large Language Models
  6. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
  7. 2025A Bayesian Hybrid Parameter-Efficient Fine-Tuning Method for Large Language Models
  8. 2024Scaling Sparse Fine-Tuning to Large Language Models