mlsanityData to 14 Oct 2026

The Thermodynamic Costs of Simple Linear Regression

arXiv 2605.19195 ↗First posted 18 May 2026cond-mat.stat-mech cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

The construction of models from data is a significant contributor to the energetic costs of computation. Because of this, understanding how foundational thermodynamic bounds apply to modeling algorithms will be increasingly important. Here, we study the thermodynamic costs of a basic and fundamental modeling algorithm: simple linear regression. Following Landauer, we approximate the thermodynamic lower bound on irreversibly performing both exact linear regression and linear regression via stochastic gradient descent as implemented on floating-point numbers. From this, we derive energycost aware scaling laws for the optimal dataset size for training a linear regression model given a generalization error dependent demand for inference. Additionally, we discuss a method to lower bound the entropy production from the mismatch cost for algorithms with continuous input variables.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 86 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

27 of 86 references.

References · 27

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  8. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  10. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  11. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  12. 2018To understand deep learning we need to understand kernel learningcited by 187
15 more
  1. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  2. 2023Microscaling Data Formats for Deep Learningcited by 152
  3. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  4. 2022FP8 Quantization: The Power of the Exponentcited by 88
  5. 2023A Systematic Review of Green AIcited by 77
  6. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 57
  7. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  8. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 14
  9. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniquescited by 10
  10. 2024EfQAT: An Efficient Framework for Quantization-Aware Trainingcited by 7
  11. 2025Predicting kernel regression learning curves from only raw data statisticscited by 6
  12. 2023Is stochastic thermodynamics the key to understanding the energy costs of computation?cited by 5
  13. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  14. 2021EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Modelscited by 1
  15. 2021An Efficient Reversible Algorithm for Linear Regressioncited by 1

Built on by

Not cited yet.