mlsanityData to 14 Oct 2026

Efficient Fine-Tuning of BERT Models on the Edge

arXiv 2205.01541 ↗First posted 3 May 2022cs.LG cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Resource-constrained devices are increasingly the deployment targets of machine learning applications. Static models, however, do not always suffice for dynamic environments. On-device training of models allows for quick adaptability to new scenarios. With the increasing size of deep neural networks, as noted with the likes of BERT and other natural language processing models, comes increased resource requirements, namely memory, computation, energy, and time. Furthermore, training is far more resource intensive than inference. Resource-constrained on-device learning is thus doubly difficult, especially with large BERT-like models. By reducing the memory usage of fine-tuning, pre-trained BERT models can become efficient enough to fine-tune on resource-constrained devices. We propose Freeze And Reconfigure (FAR), a memory-efficient training regime for BERT-like models that reduces the memory usage of activation maps during fine-tuning by avoiding unnecessary parameter updates. FAR reduces fine-tuning time on the DistilBERT model and CoLA dataset by 30%, and time spent on memory operations by 47%. More broadly, reductions in metric performance on the GLUE and SQuAD datasets are around 1% on average.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

17 of 23 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  11. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
5 more
  1. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  2. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  3. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  4. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  5. 2018Full deep neural network training on a pruned weight budgetcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 16; 2025: 5; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  3. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  4. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  5. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  6. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  7. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  8. 2024Large Language Model for Mental Health: A Systematic Reviewcited by 14
  9. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  10. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  11. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  12. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptationcited by 10
17 more
  1. 2024PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptationcited by 8
  2. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  3. 2024PersianMind: A Cross-Lingual Persian-English Large Language Modelcited by 7
  4. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adaptercited by 5
  5. 2025Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuningcited by 3
  6. 2022Efficient Fine-Tuning of Compressed Language Models with Learnerscited by 1
  7. 2023Sensitivity-Aware Finetuning for Accuracy Recovery on Deep Learning Hardwarecited by 1
  8. 2024CERM: Context-aware Literature-based Discovery via Sentiment Analysiscited by 1
  9. 2024Foundation Model Engineering: Engineering Foundation Models Just as Engineering Softwarecited by 1
  10. 2024AI Act for the Working Programmercited by 1
  11. 2025Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecastingcited by 1
  12. 2023Clinical BioBERT Hyperparameter Optimization using Genetic Algorithmcited by 0
  13. 2023Subspace-Configurable Networkscited by 0
  14. 2024HUT: A More Computation Efficient Fine-Tuning Method With Hadamard Updated Transformationcited by 0
  15. 2025Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Modelscited by 0
  16. 2025AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Modelscited by 0
  17. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0