# Holistically Evaluating the Environmental Impact of Creating Language Models

arXiv: https://arxiv.org/abs/2503.05804 · HTML: https://mlsanity.com/paper/2503.05804/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [Measuring the Carbon Intensity of AI in Cloud Instances](https://mlsanity.com/paper/2206.05229/) (2022)
- [OpenELM: An Efficient Language Model Family with Open Training and Inference Framework](https://mlsanity.com/paper/2404.14619/) (2024)
- [Making AI Less "Thirsty": Uncovering and Addressing the Secret Water Footprint of AI Models](https://mlsanity.com/paper/2304.03271/) (2023)
- [Establishing Task Scaling Laws via Compute-Efficient Model Ladders](https://mlsanity.com/paper/2412.04403/) (2024)
- [Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation](https://mlsanity.com/paper/2307.09701/) (2023)

## Built on by (most cited first, 26 of 26)

- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation](https://mlsanity.com/paper/2511.13655/) (2025)
- [Efficient LLMs with AMP: Attention Heads and MLP Pruning](https://mlsanity.com/paper/2504.21174/) (2025)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [Not All Water Consumption Is Equal: A Water Stress Weighted Metric for Sustainable Computing](https://mlsanity.com/paper/2506.22773/) (2025)
- [Layer-wise LoRA fine-tuning: a similarity metric approach](https://mlsanity.com/paper/2602.05988/) (2026)
- [Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks](https://mlsanity.com/paper/2506.20009/) (2025)
- [Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention](https://mlsanity.com/paper/2507.07247/) (2025)
- [Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming](https://mlsanity.com/paper/2508.08332/) (2025)
- [The Environmental Impacts of Language Model Training Keep Rising Now is the Time to Catch Impacts on the Rebound](https://mlsanity.com/paper/2510.09022/) (2025)
- [Neural Neural Scaling Laws](https://mlsanity.com/paper/2601.19831/) (2026)
- [The Environmental Cost of LLMs in AIED: Reporting and Practices](https://mlsanity.com/paper/2606.11215/) (2026)
- [One Period to Rule Them All: Identifying Critical Learning Periods in Deep Networks](https://mlsanity.com/paper/2506.15954/) (2025)
- [Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models](https://mlsanity.com/paper/2509.12960/) (2025)
- [Pruning Everything, Everywhere, All at Once](https://mlsanity.com/paper/2506.04513/) (2025)
- [The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations](https://mlsanity.com/paper/2507.13302/) (2025)
- [ML-EcoLyzer: Quantifying the Environmental Cost of Machine Learning Inference Across Frameworks and Hardware](https://mlsanity.com/paper/2511.06694/) (2025)
- [Emissions and Performance Trade-off Between Small and Large Language Models](https://mlsanity.com/paper/2601.08844/) (2025)
- [Environmental Footprint of GenAI Research: Insights from the Moshi Foundation Model](https://mlsanity.com/paper/2604.11154/) (2026)
- [Transparent Screening for LLM Inference and Training Impacts](https://mlsanity.com/paper/2604.19757/) (2026)
- [Long Live the Librarian! A Persistent Search Sub-Agent for Energy-Efficient Multi-Agent Software Engineering Systems](https://mlsanity.com/paper/2605.27787/) (2026)
- [Certificates without Electrons? Theory and Evidence on Impacts from AI-Driven Power Demand](https://mlsanity.com/paper/2606.00811/) (2026)
- [Evaluation of ML Resource Utilization Requires Model Life Cycle Assessment](https://mlsanity.com/paper/2606.07632/) (2026)
- [Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumption](https://mlsanity.com/paper/2607.04553/) (2026)
- [Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension](https://mlsanity.com/paper/2608.10296/) (2026)
- [Artificial Intelligence for Energy Optimization in Data Centers](https://mlsanity.com/paper/2609.03716/) (2026)

## Abstract (arXiv)

As the performance of artificial intelligence systems has dramatically increased, so too has the environmental impact of creating these systems. While many model developers release estimates of the power consumption and carbon emissions from the final training runs for their latest models, there is comparatively little transparency into the impact of model development, hardware manufacturing, and total water usage throughout. In this work, we estimate the real-world environmental impact of developing a series of language models, ranging from 20 million to 13 billion active parameters, trained on up to 5.6 trillion tokens each. When accounting for hardware manufacturing, model development, and our final training runs, we find that our series of models released 493 metric tons of carbon emissions, equivalent to powering about 98 homes in the United States for one year, and consumed 2.769 million liters of water, equivalent to about 24.5 years of water usage by a person in the United States, even though our data center is extremely water-efficient. We measure and report the environmental impact of our model development; to the best of our knowledge we are the first to do so for LLMs, and we find that model development, the impact of which is generally not disclosed by most model developers, amounted to ~50% of that of training. By looking at detailed time series data for power consumption, we also find that power usage throughout training is not consistent, fluctuating between ~15% and ~85% of our hardware's maximum power draw, with negative implications for grid-scale planning as demand continues to grow. We close with a discussion on the continued difficulty of estimating the environmental impact of AI systems, and key takeaways for model developers and the public at large.