mlsanityData to 7 Oct 2026

Learning Invariant Weights in Neural Networks

arXiv 2202.12439 ↗First posted 25 Feb 2022stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Assumptions about invariances or symmetries in data can significantly increase the predictive power of statistical models. Many commonly used models in machine learning are constraint to respect certain symmetries in the data, such as translation equivariance in convolutional neural networks, and incorporation of new symmetry types is actively being studied. Yet, efforts to learn such invariances from the data itself remains an open research problem. It has been shown that marginal likelihood offers a principled way to learn invariances in Gaussian Processes. We propose a weight-space equivalent to this approach, by minimizing a lower bound on the marginal likelihood to learn invariances in neural networks resulting in naturally higher performing models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

28 of 43 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2015Spatial Transformer Networkscited by 1,423
  7. 2016Group Equivariant Convolutional Networkscited by 1,154
  8. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  9. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  10. 2019General E(2)-Equivariant Steerable CNNscited by 441
  11. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  12. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
16 more
  1. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  2. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  3. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  4. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  5. 2016Rotation equivariant vector field networkscited by 98
  6. 2019B-Spline CNNs on Lie Groupscited by 97
  7. 2017Polar Transformer Networkscited by 81
  8. 2020Meta-Learning Symmetries by Reparameterizationcited by 75
  9. 2017On weight initialization in deep neural networkscited by 67
  10. 2020Learning Invariances in Neural Networkscited by 52
  11. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  12. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  13. 2019On the marginal likelihood and cross-validationcited by 36
  14. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  15. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  16. 2016Local Group Invariant Representations via Orbit Embeddingscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 4; 2025: 1; 2026: 1.

16 papers.

Sort:

  1. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  2. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  3. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  4. 2023From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communicationcited by 20
  5. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  6. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  7. 2024A Complexity-Based Theory of Compositionalitycited by 13
  8. 2025Position: A Theory of Deep Learning Must Include Compositional Sparsitycited by 13
  9. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  10. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  11. 2024A Generative Model of Symmetry Transformationscited by 10
  12. 2023Using and Abusing Equivariancecited by 4
4 more
  1. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  2. 2022A tradeoff between universality of equivariant models and learnability of symmetriescited by 1
  3. 2023Uncertainty in Graph Contrastive Learning with Bayesian Neural Networkscited by 0
  4. 2026Equivariance and Augmentation for Bayesian Neural Networkscited by 0