# FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment

arXiv: https://arxiv.org/abs/2608.14550 · HTML: https://mlsanity.com/paper/2608.14550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 17 of 17)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks](https://mlsanity.com/paper/2303.03667/) (2023)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [Accelerating BERT Inference for Sequence Labeling via Early-Exit](https://mlsanity.com/paper/2105.13878/) (2021)
- [Benchmarking and Dissecting the Nvidia Hopper GPU Architecture](https://mlsanity.com/paper/2402.13499/) (2024)
- [Greening Large Language Models of Code](https://mlsanity.com/paper/2309.04076/) (2023)
- [Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation](https://mlsanity.com/paper/2309.13192/) (2023)
- [FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning](https://mlsanity.com/paper/2403.07094/) (2024)
- [Dissecting FLOPs along input dimensions for GreenAI cost estimations](https://mlsanity.com/paper/2107.11949/) (2021)
- [Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices](https://mlsanity.com/paper/2506.01774/) (2025)
- [Energy Aware Development of Neuromorphic Implantables: From Metrics to Action](https://mlsanity.com/paper/2506.09599/) (2025)

## Abstract (arXiv)

AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs. While reporting Floating Point Operations (FLOPs) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others. This paper sets out to replicate the original experiments from a study that proposed the α-FLOPs estimation formula to verify whether the results remain applicable on newer, more powerful hardware. During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data. Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions. However, fine-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that the α-FLOPs formula generally underestimates. Ultimately, this work validates the empirical findings from the original study but shows negative results when applying the α-FLOPs estimation. We also highlight the critical need for complete and accurate replication packages for research on hardware-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study.