mlsanityData to 7 Oct 2026

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

arXiv 2106.02624 ↗First posted 4 Jun 2021cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. We demonstrate this by conducting performance benchmarks and substantiate ViViT's usefulness by studying the impact of noise on the GGN's structural properties during neural network training.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 27 of its 60 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

27 of 60 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  8. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  9. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
15 more
  1. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  2. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  3. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  4. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  5. 2020The Early Phase of Neural Network Trainingcited by 114
  6. 2019BackPACK: Packing more into backpropcited by 77
  7. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  8. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  9. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  10. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  11. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  12. 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 18
  13. 2018Estimating the Spectral Density of Large Implicit Matricescited by 16
  14. 2017Block-diagonal Hessian-free Optimization for Training Neural Networkscited by 10
  15. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 0; 2024: 5; 2025: 4; 2026: 4.

16 papers.

Sort:

  1. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  2. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  3. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  4. 2025Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomizationcited by 13
  5. 2024Linearization Turns Neural Operators into Function-Valued Gaussian Processescited by 11
  6. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  7. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  8. 2022FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Datacited by 4
  9. 2024Debiasing Mini-Batch Quadratics for Applications in Deep Learningcited by 2
  10. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  11. 2025Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methodscited by 1
  12. 2024TrAct: Making First-layer Pre-Activations Trainablecited by 0
4 more
  1. 2026Inter-Layer Hessian Analysis of Neural Networks with DAG Architecturescited by 0
  2. 2026Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagationcited by 0
  3. 2026CoeF-SFL: Preserving Collaborative Server-Client Learning with Enhanced Communication Efficiencycited by 0
  4. 2026Certified Approximation for Interpretable Representer Landmarkscited by 0