# HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing

arXiv: https://arxiv.org/abs/2002.05829 · HTML: https://mlsanity.com/paper/2002.05829/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2020.

## Built on by (most cited first, 16 of 16)

- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning](https://mlsanity.com/paper/2302.08476/) (2023)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models](https://mlsanity.com/paper/2210.07111/) (2022)
- [IrEne: Interpretable Energy Prediction for Transformers](https://mlsanity.com/paper/2106.01199/) (2021)
- [The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment](https://mlsanity.com/paper/2302.06117/) (2023)
- [Energy and Carbon Considerations of Fine-Tuning BERT](https://mlsanity.com/paper/2311.10267/) (2023)
- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation](https://mlsanity.com/paper/2307.09701/) (2023)
- [Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models](https://mlsanity.com/paper/2204.07288/) (2022)
- [Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research](https://mlsanity.com/paper/2306.16900/) (2023)
- [AI, Climate, and Regulation: From Data Centers to the AI Act](https://mlsanity.com/paper/2410.06681/) (2024)
- [Conformal Predictor for Improving Zero-shot Text Classification Efficiency](https://mlsanity.com/paper/2210.12619/) (2022)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [MetaGreen: Meta-Learning Inspired Transformer Selection for Green Semantic Communication](https://mlsanity.com/paper/2406.16962/) (2024)

## Abstract (arXiv)

Computation-intensive pretrained models have been taking the lead of many natural language processing benchmarks such as GLUE. However, energy efficiency in the process of model training and inference becomes a critical bottleneck. We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing. With HULK, we compare pretrained models' energy efficiency from the perspectives of time and cost. Baseline benchmarking results are provided for further analysis. The fine-tuning efficiency of different pretrained models can differ a lot among different tasks and fewer parameter number does not necessarily imply better efficiency. We analyzed such phenomenon and demonstrate the method of comparing the multi-task efficiency of pretrained models. Our platform is available at https://sites.engineering.ucsb.edu/~xiyou/hulk/.