mlsanityData to 7 Oct 2026

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

arXiv 2106.10199 ↗First posted 18 Jun 2021cs.LG cs.CLcited by 1,234

Not on the map. Why?

Cited by 47 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Key facts. Cited by 1,234 papers in the map (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
15 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  7. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  10. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  11. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  12. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  13. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  14. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 56
  15. 2018Extreme Adaptation for Personalized Neural Machine Translationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 129; 2023: 275; 2024: 347; 2025: 293; 2026: 169.

1,234 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  11. 2024A Survey on Large Language Models for Code Generationcited by 646
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  7. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  9. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  11. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  12. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  13. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  14. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  15. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  16. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  17. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  18. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  19. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  20. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  21. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  22. 2024Generative Representational Instruction Tuningcited by 227
  23. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  24. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  25. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  26. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  27. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  28. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  29. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  30. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  31. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  32. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  33. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  34. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  35. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  36. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  37. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  38. 2023Language-Driven Representation Learning for Roboticscited by 163

50 most cited of 1,234 All 1,234 (JSON)