mlsanityData to 7 Oct 2026

Dissecting Transformers: A CLEAR Perspective towards Green AI

arXiv 2510.02810 ↗First posted 3 Oct 2025cs.LG cs.AI cs.SEcited by 1

Not on the map. Why?

Abstract (arXiv)

The rapid adoption of Large Language Models (LLMs) has raised significant environmental concerns. Unlike the one-time cost of training, LLM inference occurs continuously and dominates the AI energy footprint. Yet most sustainability studies report only coarse model-level metrics, treating energy efficiency as an afterthought rather than a primary objective. Addressing the limitation, we propose Component-Level Energy Assessment via Repetitions CLEAR, to overcome temporal mismatch between microsecond scale component execution and millisecond(ms) scale monitoring of energy sensors. Using CLEAR, we evaluate 15 models spanning four architecture types, keeping component-wise energy variance below 9.5% while capturing over 90% of total energy as individual components. We present the first comprehensive, fine-grained energy analysis of Transformer components across key parameters such as batch size, attention heads, hidden dimension, KV cache, and attention variants. Our findings reveal that Attention consumes significantly more Energy per FLOP as compared to the entire model, indicating that FLOPs alone fail to capture true component-level energy cost. CLEAR enables reliable fine-grained energy measurements and provides a strong formal foundation for predictive modelling of energy consumption.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

23 of 51 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  7. 2019Green AIcited by 620
  8. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  9. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  10. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 73
  11. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  12. 2023Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Studycited by 23
11 more
  1. 2023Part-time Power Measurements: nvidia-smi's Lack of Attentioncited by 15
  2. 2025Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulationscited by 12
  3. 2021IrEne: Interpretable Energy Prediction for Transformerscited by 11
  4. 2024Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructurescited by 11
  5. 2020Verified Instruction-Level Energy Consumption Measurement for NVIDIA GPUscited by 9
  6. 2023Accuracy is not the only Metric that matters: Estimating the Energy Consumption of Deep Learning Modelscited by 9
  7. 2023Enhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurementcited by 5
  8. 2025Green MLOps to Green GenOps: An Empirical Study of Energy Consumption in Discriminative and Generative AI Operationscited by 2
  9. 2025EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Modelcited by 2
  10. 2025Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attentioncited by 2
  11. 2024CPPJoules: An Energy Measurement Tool for C++cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Transformers to Large Language Models: A systematic review of AI applications in the energy sector towards Agentic Digital Twins
  2. 2026Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference
  3. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  4. 2025StructuredDNA: A Bio-Physical Framework for Energy-Aware Transformer Routing
  5. 2025Addressing the sustainable AI trilemma: a case study on LLM agents and RAG
  6. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architectures
  7. 2026Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models
  8. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsity