mlsanityData to 7 Oct 2026

BackPACK: Packing more into backprop

arXiv 1912.10985 ↗First posted 23 Dec 2019cs.LG stat.MLcited by 77

Not on the map. Why?

Abstract (arXiv)

Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient. Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in theory, be computed efficiently, and at the same time as the gradient. While these quantities are of great interest to researchers and practitioners, current deep-learning software does not support their automatic calculation. Manually implementing them is burdensome, inefficient if done naively, and the resulting code is rarely shared. This hampers progress in deep learning, and unnecessarily narrows research to focus on gradient descent and its variants; it also complicates replication studies and comparisons between newly developed methods that require those quantities, to the point of impossibility. To address this problem, we introduce BackPACK, an efficient framework built on top of PyTorch, that extends the backpropagation algorithm to extract additional information from first- and second-order derivatives. Its capabilities are illustrated by benchmark reports for computing additional quantities on deep neural networks, and an example application by testing several recent curvature approximations for optimization.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 13; 2023: 16; 2024: 15; 2025: 8; 2026: 7.

77 papers.

Sort:

By area:

  1. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  4. 2021Deep Learning with Label Differential Privacycited by 113
  5. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  6. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  7. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  8. 2022Uncertainty-guided Source-free Domain Adaptationcited by 46
  9. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  10. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 35
  11. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  12. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 32
38 more
  1. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  2. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  3. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  4. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  5. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  6. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  7. 2022Model-agnostic out-of-distribution detection using combined statistical testscited by 14
  8. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  9. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 13
  10. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 13
  11. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  12. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  13. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
  14. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  15. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  16. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  17. 2023Bayesian Metric Learning for Uncertainty Quantification in Image Retrievalcited by 9
  18. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  19. 2021Improving Differentially Private SGD via Randomly Sparsified Gradientscited by 8
  20. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  21. 2022HesScale: Scalable Computation of Hessian Diagonalscited by 8
  22. 2023Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noisecited by 8
  23. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  24. 2021TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversioncited by 7
  25. 2021Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimationcited by 7
  26. 2022Adaptive Learning Rates for Faster Stochastic Gradient Methodscited by 7
  27. 2024Unknown Domain Inconsistency Minimization for Domain Generalizationcited by 7
  28. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  29. 2023ISAAC Newton: Input-based Approximate Curvature for Newton's Methodcited by 6
  30. 2023Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory Distillationcited by 6
  31. 2024Variance reduction of diffusion model's gradients with Taylor approximation-based control variatecited by 6
  32. 2023Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspectivecited by 5
  33. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  34. 2023Learning Capacity: A Measure of the Effective Dimensionality of a Modelcited by 4
  35. 2024Batch size invariant Adamcited by 4
  36. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  37. 2022Cold Posteriors through PAC-Bayescited by 3
  38. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3

50 most cited of 77 All 77 (JSON)