mlsanityData to 14 Oct 2026

PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark

arXiv 2511.21285 ↗First posted 26 Nov 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite the state-of-the-art performance of Large Language Models (LLMs) achieved on many tasks, their massive scale often leads to high computational and environmental costs, limiting their accessibility. Parameter-Efficient Fine-Tuning (PEFT) methods address this challenge by reducing the number of trainable parameters while maintaining strong downstream performance. Despite the advances in PEFT methods, current evaluations remain limited (in terms of evaluated models and datasets) and difficult to reproduce. To bridge this gap, we introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs. We demonstrate its usage across 27 NLP datasets and 7 PEFT methods. To account for different PEFT training and inference factors, we also introduce the PEFT Soft Cost Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

37 of 71 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
25 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  12. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  13. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  14. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  15. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  16. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  17. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  18. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  19. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Modelscited by 16
  20. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  21. 2025Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspacecited by 11
  22. 2019What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answeringcited by 7
  23. 2025Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuningcited by 6
  24. 2025Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Productcited by 3
  25. 2023ZhiJian: A Unifying and Rapidly Deployable Toolbox for Pre-trained Model Reusecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  2. 2026Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Modelscited by 2
  3. 2025PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Modelscited by 0
  4. 2026Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shiftcited by 0
  5. 2026Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?cited by 0