# Energy and Carbon Considerations of Fine-Tuning BERT

arXiv: https://arxiv.org/abs/2311.10267 · HTML: https://mlsanity.com/paper/2311.10267/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [Measuring the Carbon Intensity of AI in Cloud Instances](https://mlsanity.com/paper/2206.05229/) (2022)
- [HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing](https://mlsanity.com/paper/2002.05829/) (2020)
- [IrEne: Interpretable Energy Prediction for Transformers](https://mlsanity.com/paper/2106.01199/) (2021)
- [Towards Quantifying the Carbon Emissions of Differentially Private Machine Learning](https://mlsanity.com/paper/2107.06946/) (2021)
- [Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models](https://mlsanity.com/paper/2204.07288/) (2022)

## Built on by (most cited first, 11 of 11)

- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [LUMIA: Linear probing for Unimodal and MultiModal Membership Inference Attacks leveraging internal LLM states](https://mlsanity.com/paper/2411.19876/) (2024)
- [LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs](https://mlsanity.com/paper/2505.24451/) (2025)
- [SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version](https://mlsanity.com/paper/2508.15478/) (2025)
- [Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solvers](https://mlsanity.com/paper/2505.12556/) (2025)
- [Eco-Aware Graph Neural Networks for Sustainable Recommendations](https://mlsanity.com/paper/2410.09514/) (2024)
- [Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languages](https://mlsanity.com/paper/2508.06435/) (2025)
- [XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML](https://mlsanity.com/paper/2508.00924/) (2025)
- [Rethinking the Harmonic Loss via Non-Euclidean Distance Layers](https://mlsanity.com/paper/2603.10225/) (2026)
- [Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languages](https://mlsanity.com/paper/2607.12612/) (2026)

## Abstract (arXiv)

Despite the popularity of the `pre-train then fine-tune' paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.