mlsanityData to 7 Oct 2026

Learning Layer-wise Equivariances Automatically using Gradients

arXiv 2310.06131 ↗First posted 9 Oct 2023cs.LG cs.AI stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. However, symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients. Learning symmetry and associated weight connectivity structures from scratch is difficult for two reasons. First, it requires efficient and flexible parameterisations of layer-wise equivariances. Secondly, symmetries act as constraints and are therefore not encouraged by training losses measuring data fit. To overcome these challenges, we improve parameterisations of soft equivariance and learn the amount of equivariance in layers by optimising the marginal likelihood, estimated using differentiable Laplace approximations. The objective balances data fit and model complexity enabling layer-wise symmetry discovery in deep networks. We demonstrate the ability to automatically learn layer-wise equivariances on image classification tasks, achieving equivalent or improved performance over baselines with hard-coded symmetry.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 55 of its 70 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

55 of 70 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2017Deep Setscited by 1,394
  6. 2016Group Equivariant Convolutional Networkscited by 1,154
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  8. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  11. 2017Dynamic Routing Between Capsulescited by 689
  12. 2019General E(2)-Equivariant Steerable CNNscited by 441
38 more
  1. 2014New insights and perspectives on the natural gradient methodcited by 435
  2. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  3. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  4. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  5. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  6. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  7. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  8. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  9. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  10. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  11. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  12. 2020Incorporating Symmetry into Deep Dynamics Models for Improved Generalizationcited by 123
  13. 2021Bayesian Neural Network Priors Revisitedcited by 121
  14. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  15. 2019B-Spline CNNs on Lie Groupscited by 97
  16. 2022Approximately Equivariant Networks for Imperfectly Symmetric Dynamicscited by 95
  17. 2019Stacked Capsule Autoencoderscited by 82
  18. 2021Automatic Symmetry Discovery with Lie Algebra Convolutional Networkcited by 82
  19. 2020Meta-Learning Symmetries by Reparameterizationcited by 75
  20. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  21. 2019BayesNAS: A Bayesian Approach for Neural Architecture Searchcited by 67
  22. 2020Depth Uncertainty in Neural Networkscited by 56
  23. 2020Truly shift-invariant convolutional neural networkscited by 53
  24. 2020Learning Invariances in Neural Networkscited by 52
  25. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  26. 2020Towards Learning Convolutions from Scratchcited by 46
  27. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  28. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  29. 2014Learning the Irreducible Representations of Commutative Lie Groupscited by 41
  30. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  31. 2023Generative Adversarial Symmetry Discoverycited by 40
  32. 2022Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact casecited by 25
  33. 2021Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groupscited by 22
  34. 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
  35. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  36. 2022Equivariance Discovery by Learned Parameter-Sharingcited by 18
  37. 2018Intertwiners between Induced Representations (with Applications to the Theory of Equivariant Neural Networks)cited by 16
  38. 2022Learning Invariant Weights in Neural Networkscited by 16

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 6; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2023The LLM Surgeoncited by 36
  3. 2024Symmetry-Informed Governing Equation Discoverycited by 14
  4. 2024Relaxed Equivariance via Multitask Learningcited by 13
  5. 2024Ensembles provably learn equivariance through data augmentationcited by 8
  6. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  7. 2024Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networkscited by 7
  8. 2024Symmetry Discovery for Different Data Typescited by 7
  9. 2025Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetrycited by 6
  10. 2024Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensorscited by 5
  11. 2025laplax -- Laplace Approximations with JAXcited by 5
  12. 2025Explicit Discovery of Nonlinear Symmetries from Dynamic Datacited by 5
11 more
  1. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  2. 2025AtlasD: Automatic Local Symmetry Discoverycited by 4
  3. 2026Data Augmentation: A Fourier Analysis Perspectivecited by 3
  4. 2026Adaptive Symmetry Discovery for Dynamical System Identificationcited by 2
  5. 2026Efficient Learning and Symmetry Discovery under Exact Invariancescited by 2
  6. 2025Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusioncited by 1
  7. 2025On Equivariant Model Selection through the Lens of Uncertaintycited by 1
  8. 2026Scale-invariant Gaussian derivative residual networkscited by 0
  9. 2026Tunable Soft Equivariance with Guaranteescited by 0
  10. 2026Sparse Data Augmentation for Optimization with Provable Guaranteescited by 0
  11. 2026LieDiscover: Adaptive Symbolic Library Construction for Explicit Open-form Symmetry Discoverycited by 0