mlsanityData to 7 Oct 2026

Equivariance-aware Architectural Optimization of Neural Networks

arXiv 2210.05484 ↗First posted 11 Oct 2022cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Incorporating equivariance to symmetry groups as a constraint during neural network training can improve performance and generalization for tasks exhibiting those symmetries, but such symmetries are often not perfectly nor explicitly present. This motivates algorithmically optimizing the architectural constraints imposed by equivariance. We propose the equivariance relaxation morphism, which preserves functionality while reparameterizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the [G]-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization. We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization. Experiments across a variety of datasets show the benefit of dynamically constrained equivariance to find effective architectures with approximate equivariance.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

31 of 42 references.

References · 31

  1. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2016Group Equivariant Convolutional Networkscited by 1,154
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  5. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  6. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  7. 2019General E(2)-Equivariant Steerable CNNscited by 441
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  9. 2017Large-Scale Evolution of Image Classifierscited by 348
  10. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  11. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 224
  12. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
19 more
  1. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  2. 2017Equivariance Through Parameter-Sharingcited by 139
  3. 2022Approximately Equivariant Networks for Imperfectly Symmetric Dynamicscited by 95
  4. 2020Meta-Learning Symmetries by Reparameterizationcited by 75
  5. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  6. 2021Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifoldscited by 72
  7. 2016Network Morphismcited by 63
  8. 2020Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gapcited by 50
  9. 2021Learning Partial Equivariances from Datacited by 44
  10. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  11. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  12. 2019Efficient Neural Architecture Search via Proximal Iterationscited by 31
  13. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 31
  14. 2020Revisiting Spatial Invariance with Low-Rank Local Connectivitycited by 29
  15. 2022Equivariance Discovery by Learned Parameter-Sharingcited by 18
  16. 2022A Classification of G-invariant Shallow Neural Networkscited by 5
  17. 2022POPNASv2: An Efficient Multi-Objective Neural Architecture Search Techniquecited by 5
  18. 2021Autoequivariant Network Search via Group Decompositioncited by 3
  19. 2022Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 3.

8 papers.

  1. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  2. 2023A General Theory of Correct, Incorrect, and Extrinsic Equivariancecited by 18
  3. 2023Efficient Equivariant Transfer Learning from Pretrained Modelscited by 11
  4. 2024A Generative Model of Symmetry Transformationscited by 10
  5. 2024Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networkscited by 7
  6. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  7. 2023Densely Connected G-invariant Deep Neural Networks with Signed Permutation Representationscited by 2
  8. 2023Efficient Model-Agnostic Multi-Group Equivariant Networkscited by 1