Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
Not on the map. Why?
Abstract (arXiv)
Parameter-Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT) while requiring significantly fewer computing resources, making it the go-to choice for researchers. We find that although PEFT can achieve competitive results on some benchmarks, its performance falls short of FFT in complex tasks, such as reasoning and instruction-based fine-tuning. In this paper, we compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory. We theoretically demonstrate that PEFT is a strict subset of FFT. By providing theoretical upper bounds for PEFT, we show that the limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories. We hope that these results spark further research beyond the realms of well established PEFT. The source code is in the anonymous Github repositoryfootnotehttps://github.com/misonsky/PEFTEval.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
46 of 69 references.
References · 46
- 2017Attention Is All You Needcited by 59,932
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Mistral 7Bcited by 3,065
34 more
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
- 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
- 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
- 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
- 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
- 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
- 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
- 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
- 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
- 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
- 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
- 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
- 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
- 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
- 2022Rank Diminishing in Deep Neural Networkscited by 47
- 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
- 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
- 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
- 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
- 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
- 2024Parameter Efficient Instruction Tuning: An Empirical Studycited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.