mlsanityData to 7 Oct 2026

ISAAC Newton: Input-based Approximate Curvature for Newton's Method

arXiv 2305.00604 ↗First posted 1 May 2023cs.LG cs.CV math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 24 of its 42 references that are in the map. First posted on arXiv 1 May 2023.

Builds on

24 of 42 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2014New insights and perspectives on the natural gradient methodcited by 435
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  11. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  12. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
12 more
  1. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  2. 2019BackPACK: Packing more into backpropcited by 77
  3. 2015Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergencecited by 72
  4. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  5. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  6. 2015Convergence rates of sub-sampled Newton methodscited by 32
  7. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
  8. 2016Sub-sampled Newton Methods with Non-uniform Samplingcited by 21
  9. 2019Modular Block-diagonal Curvature Approximations for Feedforward Architecturescited by 13
  10. 2016Sub-Sampled Newton Methods I: Globally Convergent Algorithmscited by 8
  11. 2015Faster SGD Using Sketched Conditioningcited by 3
  12. 2020Convex optimization based on global lower second-order modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 1.

6 papers.

  1. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  2. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  3. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  4. 2024Newton Losses: Using Curvature Information for Learning with Differentiable Algorithmscited by 2
  5. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  6. 2024TrAct: Making First-layer Pre-Activations Trainablecited by 0