# Beyond Efficiency: Scaling AI Sustainably

arXiv: https://arxiv.org/abs/2406.05303 · HTML: https://mlsanity.com/paper/2406.05303/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [Chasing Carbon: The Elusive Environmental Footprint of Computing](https://mlsanity.com/paper/2011.02839/) (2020)
- [An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage](https://mlsanity.com/paper/2010.09435/) (2020)
- [Carbon-Aware Computing for Datacenters](https://mlsanity.com/paper/2106.11750/) (2021)
- [Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenters](https://mlsanity.com/paper/2201.10036/) (2022)
- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [SVP-CF: Selection via Proxy for Collaborative Filtering Data](https://mlsanity.com/paper/2107.04984/) (2021)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Treehouse: A Case For Carbon-Aware Datacenter Software](https://mlsanity.com/paper/2201.02120/) (2022)
- [Summarizing CPU and GPU Design Trends with Product Data](https://mlsanity.com/paper/1911.11313/) (2019)
- [Adapting Datacenter Capacity for Greener Datacenters and Grid](https://mlsanity.com/paper/2301.03148/) (2023)
- [Generative AI Beyond LLMs: System Implications of Multi-Modal Generation](https://mlsanity.com/paper/2312.14385/) (2023)
- [Carbon Responder: Coordinating Demand Response for the Datacenter Fleet](https://mlsanity.com/paper/2311.08589/) (2023)
- [Accelerator-level Parallelism](https://mlsanity.com/paper/1907.02064/) (2019)
- [Socio-Technological Challenges and Opportunities: Paths Forward](https://mlsanity.com/paper/2108.06738/) (2021)
- [Unlocking the Potential of Renewable Energy Through Curtailment Prediction](https://mlsanity.com/paper/2405.18526/) (2024)

## Built on by (most cited first, 8 of 8)

- [Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training](https://mlsanity.com/paper/2411.13055/) (2024)
- [Position: AI Scaling: From Up to Down and Out](https://mlsanity.com/paper/2502.01677/) (2025)
- [The Energy Cost of Artificial Intelligence Lifecycle in Communication Networks](https://mlsanity.com/paper/2408.00540/) (2024)
- [Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs](https://mlsanity.com/paper/2510.18245/) (2025)
- [Characterizing and Efficiently Accelerating Multimodal Generation Model Inference](https://mlsanity.com/paper/2410.00215/) (2024)
- [Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs](https://mlsanity.com/paper/2506.08727/) (2025)
- [Artificial Intelligence in Open Source Software Engineering: A Foundation for Sustainability](https://mlsanity.com/paper/2602.07071/) (2026)
- [Artificial Intelligence for Energy Optimization in Data Centers](https://mlsanity.com/paper/2609.03716/) (2026)

## Abstract (arXiv)

Barroso's seminal contributions in energy-proportional warehouse-scale computing launched an era where modern datacenters have become more energy efficient and cost effective than ever before. At the same time, modern AI applications have driven ever-increasing demands in computing, highlighting the importance of optimizing efficiency across the entire deep learning model development cycle. This paper characterizes the carbon impact of AI, including both operational carbon emissions from training and inference as well as embodied carbon emissions from datacenter construction and hardware manufacturing. We highlight key efficiency optimization opportunities for cutting-edge AI technologies, from deep learning recommendation models to multi-modal generative AI tasks. To scale AI sustainably, we must also go beyond efficiency and optimize across the life cycle of computing infrastructures, from hardware manufacturing to datacenter operations and end-of-life processing for the hardware.