mlsanityData to 7 Oct 2026

Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations

arXiv 2202.10638 ↗First posted 22 Feb 2022stat.ML cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Data augmentation is commonly applied to improve performance of deep learning by enforcing the knowledge that certain transformations on the input preserve the output. Currently, the data augmentation parameters are chosen by human effort and costly cross-validation, which makes it cumbersome to apply to new datasets. We develop a convenient gradient-based method for selecting the data augmentation without validation data during training of a deep neural network. Our approach relies on phrasing data augmentation as an invariance in the prior distribution on the functions of a neural network, which allows us to learn it using Bayesian model selection. This has been shown to work in Gaussian processes, but not yet for deep neural networks. We propose a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective, which can be optimised without human supervision or validation data. We show that our method can successfully recover invariances present in the data, and that this improves generalisation and data efficiency on image datasets.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 48 of its 84 references that are in the map. First posted on arXiv 22 Feb 2022.

Builds on

48 of 84 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2015Spatial Transformer Networkscited by 1,423
  7. 2016Group Equivariant Convolutional Networkscited by 1,154
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  10. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  11. 2015Weight Uncertainty in Neural Networkscited by 554
  12. 2018Spherical CNNscited by 452
36 more
  1. 2015Deep Kernel Learningcited by 440
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  4. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  5. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  6. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  7. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 225
  8. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  9. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  10. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 168
  11. 2019Practical Deep Learning with Bayesian Principlescited by 158
  12. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  13. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  14. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  15. 2021Bayesian Neural Network Priors Revisitedcited by 121
  16. 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
  17. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  18. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  19. 2018Roto-Translation Covariant Convolutional Networks for Medical Image Analysiscited by 91
  20. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  21. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  22. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  23. 2014Manifold Gaussian Processes for Regressioncited by 79
  24. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  25. 2019BackPACK: Packing more into backpropcited by 77
  26. 2020Meta-Learning Symmetries by Reparameterizationcited by 75
  27. 2020Learning Invariances in Neural Networkscited by 52
  28. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  29. 2020Towards Learning Convolutions from Scratchcited by 46
  30. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  31. 2019On the marginal likelihood and cross-validationcited by 36
  32. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  33. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  34. 2020Scalable and Practical Natural Gradient for Large-Scale Deep Learningcited by 15
  35. 2021Probing as Quantifying Inductive Biascited by 11
  36. 2019A deterministic and computable Bernstein-von Mises theoremcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 19; 2024: 9; 2025: 6; 2026: 1.

42 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  3. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  4. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  5. 2023The LLM Surgeoncited by 36
  6. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  7. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  8. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  9. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  10. 2023From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communicationcited by 20
  11. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  12. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
30 more
  1. 2022Automatic Data Augmentation via Invariance-Constrained Learningcited by 17
  2. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  3. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  4. 2023Resilient Constrained Learningcited by 15
  5. 2024A Complexity-Based Theory of Compositionalitycited by 13
  6. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
  7. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  8. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  9. 2024A Generative Model of Symmetry Transformationscited by 10
  10. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  11. 2023Amortised Invariance Learning for Contrastive Self-Supervisioncited by 8
  12. 2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
  13. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  14. 2025QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edgecited by 7
  15. 2024Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensorscited by 5
  16. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  17. 2023Curvature-Sensitive Predictive Coding with Approximate Laplace Monte Carlocited by 3
  18. 2023Online Laplace Model Selection Revisitedcited by 2
  19. 2023Learning to Transform for Generalizable Instance-wise Invariancecited by 2
  20. 2023Efficient Rotation Invariance in Deep Neural Networks through Artificial Mental Rotationcited by 2
  21. 2025InvZW: Invariant Feature Learning via Noise-Adversarial Training for Robust Image Zero-Watermarkingcited by 2
  22. 2025Text-Guided Image Invariant Feature Learning for Robust Image Watermarkingcited by 1
  23. 2025Optimizing Data Augmentation through Bayesian Model Selectioncited by 1
  24. 2025Flatness After All?cited by 1
  25. 2025On Equivariant Model Selection through the Lens of Uncertaintycited by 1
  26. 2022HyperInvariances: Amortizing Invariance Learningcited by 0
  27. 2023Unveiling Invariances via Neural Network Pruningcited by 0
  28. 2023Uncertainty in Graph Contrastive Learning with Bayesian Neural Networkscited by 0
  29. 2024Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validationcited by 0
  30. 2026Sparse Data Augmentation for Optimization with Provable Guaranteescited by 0