mlsanityData to 14 Oct 2026

LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models

arXiv 2309.14393 ↗First posted 25 Sep 2023cs.CL cs.AI cs.CYcited by 72

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The carbon footprint associated with large language models (LLMs) is a significant concern, encompassing emissions from their training, inference, experimentation, and storage processes, including operational and embodied carbon emissions. An essential aspect is accurately estimating the carbon impact of emerging LLMs even before their training, which heavily relies on GPU usage. Existing studies have reported the carbon footprint of LLM training, but only one tool, mlco2, can predict the carbon footprint of new neural networks prior to physical training. However, mlco2 has several serious limitations. It cannot extend its estimation to dense or mixture-of-experts (MoE) LLMs, disregards critical architectural parameters, focuses solely on GPUs, and cannot model embodied carbon footprints. Addressing these gaps, we introduce textitcarb, an end-to-end carbon footprint projection model designed for both dense and MoE LLMs. Compared to mlco2, carb~significantly enhances the accuracy of carbon footprint estimations for various LLMs. The source code is released at urlhttps://github.com/SotaroKaneda/MLCarbon.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 35 of its 51 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

35 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2023PaLM 2 Technical Reportcited by 1,162
18 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  7. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  8. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  9. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  10. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
  11. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
  12. 2022Broken Neural Scaling Lawscited by 109
  13. 2020Chasing Carbon: The Elusive Environmental Footprint of Computingcited by 72
  14. 2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35
  15. 2022Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenterscited by 32
  16. 2022The Dirty Secret of SSDs: Embodied Carboncited by 6
  17. 2018The carbon footprint of distributed cloud storagecited by 4
  18. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelismcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 31; 2026: 13.

72 papers.

Sort:

By area:

  1. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  2. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  3. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 23
  5. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  6. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  7. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  8. 2024LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferencescited by 19
  9. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  10. 2024Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMscited by 14
  11. 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
  12. 2024Towards Sustainable Large Language Model Servingcited by 12
38 more
  1. 2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
  2. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10
  3. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  4. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  5. 2024LLeMpower: Understanding Disparities in the Control and Access of Large Language Modelscited by 5
  6. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  7. 2025EnsembleCI: Ensemble Learning for Carbon Intensity Forecastingcited by 5
  8. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  9. 2024The Energy Cost of Artificial Intelligence Lifecycle in Communication Networkscited by 4
  10. 2024Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learningcited by 4
  11. 2024On the Ethical Considerations of Generative Agentscited by 4
  12. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  13. 2025Carbon Footprint Evaluation of Code Generation through LLM as a Servicecited by 4
  14. 2025The Future of Generative AI in Software Engineering: A Vision from Industry and Academia in the European GENIUS Projectcited by 4
  15. 2024The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depthcited by 3
  16. 2025Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inferencecited by 3
  17. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  18. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  19. 2025Pruning Laws for Large Language Modelscited by 2
  20. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2
  21. 2025Throttling Web Agents Using Reasoning Gatescited by 2
  22. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  23. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  24. 2026The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Productscited by 2
  25. 2024OpenCarbonEval: A Unified Carbon Emission Estimation Framework in Large-Scale AI Modelscited by 1
  26. 2024Impact of Model Size on Fine-tuned LLM Performance in Data-to-Text Generation: A State-of-the-Art Investigationcited by 1
  27. 2024Double Jeopardy and Climate Impact in the Use of Large Language Models: Socio-economic Disparities and Reduced Utility for Non-English Speakerscited by 1
  28. 2025Addressing the sustainable AI trilemma: a case study on LLM agents and RAGcited by 1
  29. 2025One Search Fits All: Pareto-Optimal Eco-Friendly Model Selectioncited by 1
  30. 2025How Do Companies Manage the Environmental Sustainability of AI? An Interview Study About Green AI Efforts and Regulationscited by 1
  31. 2025Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMscited by 1
  32. 2025One Period to Rule Them All: Identifying Critical Learning Periods in Deep Networkscited by 1
  33. 2025Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysiscited by 1
  34. 2025Hierarchical Graph Networks for Accurate Weather Forecasting via Lightweight Trainingcited by 1
  35. 2023Holistic analysis on the sustainability of Federated Learning across AI product lifecyclecited by 0
  36. 2024IoTCO2: Assessing the End-To-End Carbon Footprint of Internet-of-Things-Enabled Deep Learningcited by 0
  37. 2024Simplicity within biological complexitycited by 0
  38. 2024Layer Pruning with Consensus: A Triple-Win Solutioncited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights
  2. 2026Wasted large language models: A life cycle thinking approach
  3. 2026Environmental Footprint of GenAI Research: Insights from the Moshi Foundation Model
  4. 2025Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
  5. 2024CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs
  6. 2022The Ecological Footprint of Neural Machine Translation Systems
  7. 2025Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View
  8. 2024Carbon Footprint Accounting Driven by Large Language Models and Retrieval-augmented Generation