# The Thermodynamic Costs of Simple Linear Regression

arXiv: https://arxiv.org/abs/2605.19195 · HTML: https://mlsanity.com/paper/2605.19195/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 86 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 27 of 27)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [A Systematic Review of Green AI](https://mlsanity.com/paper/2301.11047/) (2023)
- [A Variational Analysis of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1602.02666/) (2016)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks](https://mlsanity.com/paper/2410.20650/) (2024)
- [Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques](https://mlsanity.com/paper/2411.06084/) (2024)
- [EfQAT: An Efficient Framework for Quantization-Aware Training](https://mlsanity.com/paper/2411.11038/) (2024)
- [Predicting kernel regression learning curves from only raw data statistics](https://mlsanity.com/paper/2510.14878/) (2025)
- [Is stochastic thermodynamics the key to understanding the energy costs of computation?](https://mlsanity.com/paper/2311.17166/) (2023)
- [Characterization and Mitigation of Training Instabilities in Microscaling Formats](https://mlsanity.com/paper/2506.20752/) (2025)
- [EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Models](https://mlsanity.com/paper/2102.02705/) (2021)
- [An Efficient Reversible Algorithm for Linear Regression](https://mlsanity.com/paper/2110.14717/) (2021)

## Abstract (arXiv)

The construction of models from data is a significant contributor to the energetic costs of computation. Because of this, understanding how foundational thermodynamic bounds apply to modeling algorithms will be increasingly important. Here, we study the thermodynamic costs of a basic and fundamental modeling algorithm: simple linear regression. Following Landauer, we approximate the thermodynamic lower bound on irreversibly performing both exact linear regression and linear regression via stochastic gradient descent as implemented on floating-point numbers. From this, we derive energycost aware scaling laws for the optimal dataset size for training a linear regression model given a generalization error dependent demand for inference. Additionally, we discuss a method to lower bound the entropy production from the mismatch cost for algorithms with continuous input variables.