mlsanityData to 14 Oct 2026

Energy and Carbon Considerations of Fine-Tuning BERT

arXiv 2311.10267 ↗First posted 17 Nov 2023cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite the popularity of the `pre-train then fine-tune' paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

15 of 24 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2019Green AIcited by 627
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  9. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  10. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  11. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
  12. 2020HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processingcited by 16
3 more
  1. 2021IrEne: Interpretable Energy Prediction for Transformerscited by 11
  2. 2021Towards Quantifying the Carbon Emissions of Differentially Private Machine Learningcited by 8
  3. 2022Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 2.

11 papers.

  1. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 55
  2. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  3. 2024LUMIA: Linear probing for Unimodal and MultiModal Membership Inference Attacks leveraging internal LLM statescited by 10
  4. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
  5. 2025SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Versioncited by 3
  6. 2025Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solverscited by 2
  7. 2024Eco-Aware Graph Neural Networks for Sustainable Recommendationscited by 1
  8. 2025Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languagescited by 1
  9. 2025XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoMLcited by 0
  10. 2026Rethinking the Harmonic Loss via Non-Euclidean Distance Layerscited by 0
  11. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0