# Dissecting Transformers: A CLEAR Perspective towards Green AI

arXiv: https://arxiv.org/abs/2510.02810 · HTML: https://mlsanity.com/paper/2510.02810/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models](https://mlsanity.com/paper/2007.03051/) (2020)
- [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study](https://mlsanity.com/paper/2305.11164/) (2023)
- [Part-time Power Measurements: nvidia-smi's Lack of Attention](https://mlsanity.com/paper/2312.02741/) (2023)
- [Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations](https://mlsanity.com/paper/2507.11417/) (2025)
- [IrEne: Interpretable Energy Prediction for Transformers](https://mlsanity.com/paper/2106.01199/) (2021)
- [Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures](https://mlsanity.com/paper/2402.13640/) (2024)
- [Verified Instruction-Level Energy Consumption Measurement for NVIDIA GPUs](https://mlsanity.com/paper/2002.07795/) (2020)
- [Accuracy is not the only Metric that matters: Estimating the Energy Consumption of Deep Learning Models](https://mlsanity.com/paper/2304.00897/) (2023)
- [Enhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurement](https://mlsanity.com/paper/2308.12264/) (2023)
- [Green MLOps to Green GenOps: An Empirical Study of Energy Consumption in Discriminative and Generative AI Operations](https://mlsanity.com/paper/2503.23934/) (2025)
- [EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model](https://mlsanity.com/paper/2506.09061/) (2025)
- [Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention](https://mlsanity.com/paper/2507.07247/) (2025)
- [CPPJoules: An Energy Measurement Tool for C++](https://mlsanity.com/paper/2412.13555/) (2024)

## Built on by (most cited first, 1 of 1)

- [Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference](https://mlsanity.com/paper/2607.09520/) (2026)

## Abstract (arXiv)

The rapid adoption of Large Language Models (LLMs) has raised significant environmental concerns. Unlike the one-time cost of training, LLM inference occurs continuously and dominates the AI energy footprint. Yet most sustainability studies report only coarse model-level metrics, treating energy efficiency as an afterthought rather than a primary objective. Addressing the limitation, we propose Component-Level Energy Assessment via Repetitions CLEAR, to overcome temporal mismatch between microsecond scale component execution and millisecond(ms) scale monitoring of energy sensors. Using CLEAR, we evaluate 15 models spanning four architecture types, keeping component-wise energy variance below 9.5% while capturing over 90% of total energy as individual components. We present the first comprehensive, fine-grained energy analysis of Transformer components across key parameters such as batch size, attention heads, hidden dimension, KV cache, and attention variants. Our findings reveal that Attention consumes significantly more Energy per FLOP as compared to the entire model, indicating that FLOPs alone fail to capture true component-level energy cost. CLEAR enables reliable fine-grained energy measurements and provides a strong formal foundation for predictive modelling of energy consumption.