mlsanityData to 7 Oct 2026

Determinant Estimation under Memory Constraints and Neural Scaling Laws

arXiv 2503.04424 ↗First posted 6 Mar 2025stat.ML cs.LG cs.NAcited by 3

Not on the map. Why?

Abstract (arXiv)

Calculating or accurately estimating log-determinants of large positive definite matrices is of fundamental importance in many machine learning tasks. While its cubic computational complexity can already be prohibitive, in modern applications, even storing the matrices themselves can pose a memory bottleneck. To address this, we derive a novel hierarchical algorithm based on block-wise computation of the LDL decomposition for large-scale log-determinant calculation in memory-constrained settings. In extreme cases where matrices are highly ill-conditioned, accurately computing the full matrix itself may be infeasible. This is particularly relevant when considering kernel matrices at scale, including the empirical Neural Tangent Kernel (NTK) of neural networks trained on large datasets. Under the assumption of neural scaling laws in the test error, we show that the ratio of pseudo-determinants satisfies a power-law relationship, allowing us to derive corresponding scaling laws. This enables accurate estimation of NTK log-determinants from a tiny fraction of the full dataset; in our experiments, this results in a ~100,000× speedup with improved accuracy over competing approximations. Using these techniques, we successfully estimate log-determinants for dense matrices of extreme sizes, which were previously deemed intractable and inaccessible due to their enormous scale and computational demands.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 68 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

31 of 68 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  8. 2021Explaining Neural Scaling Lawscited by 353
  9. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  10. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  11. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  12. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
19 more
  1. 2024AI and Memory Wallcited by 89
  2. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  3. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  4. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  5. 2017Scalable Log Determinants for Gaussian Process Kernel Learningcited by 34
  6. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  7. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  8. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  9. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  10. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  11. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  12. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  13. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  14. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  15. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  16. 2017Bayesian Inference of Log Determinantscited by 4
  17. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  18. 2022TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Modelscited by 3
  19. 2022A Singular Woodbury and Pseudo-Determinant Matrix Identities and Application to Gaussian Process Regressioncited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  2. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
  3. 2025Spectral Estimation with Free Decompressioncited by 0