Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
Not on the map. Why?
Abstract (arXiv)
Equivariance has emerged as a desirable property of representations of objects subject to identity-preserving transformations that constitute a group, such as translations and rotations. However, the expressivity of a representation constrained by group equivariance is still not fully understood. We address this gap by providing a generalization of Cover's Function Counting Theorem that quantifies the number of linearly separable and group-invariant binary dichotomies that can be assigned to equivariant representations of objects. We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling. While other operations do not change the fraction of separable dichotomies, local pooling decreases the fraction, despite being a highly nonlinear operation. Finally, we test our theory on intermediate representations of randomly initialized and fully trained convolutional neural networks and find perfect agreement.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 98 references that are in the map. First posted on arXiv 14 Oct 2021.
Builds on
45 of 98 references.
References · 45
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2017Deep Setscited by 1,394
- 2016Group Equivariant Convolutional Networkscited by 1,154
- 2020Directional Message Passing for Molecular Graphscited by 556
- 2019General E(2)-Equivariant Steerable CNNscited by 441
- 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
- 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
- 2018Invariant and Equivariant Graph Networkscited by 339
- 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
- 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
- 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 262
- 2019Cormorant: Covariant Molecular Neural Networkscited by 262
33 more
- 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
- 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244
- 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
- 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 193
- 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
- 2019Universal Invariant and Equivariant Graph Neural Networkscited by 182
- 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 168
- 2019On the Universality of Invariant Networkscited by 151
- 2018Universal approximations of invariant maps by neural networkscited by 127
- 2020Lorentz Group Equivariant Neural Network for Particle Physicscited by 120
- 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
- 2019Scale-Equivariant Steerable Networkscited by 110
- 2020On Learning Sets of Symmetric Elementscited by 97
- 2018Roto-Translation Covariant Convolutional Networks for Medical Image Analysiscited by 91
- 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 90
- 2020On the Benefits of Invariance in Neural Networkscited by 78
- 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
- 2021Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifoldscited by 72
- 2018Deep Models of Interactions Across Setscited by 68
- 2017Classification and Geometry of General Perceptual Manifoldscited by 62
- 2018DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applicationscited by 61
- 2018CubeNet: Equivariance to 3D Rotation and Translationcited by 59
- 2018Covariant Compositional Networks For Learning Graphscited by 54
- 2020Universal Equivariant Multilayer Perceptronscited by 49
- 2019On Universal Equivariant Set Networkscited by 45
- 2016Generalization Error of Invariant Classifierscited by 42
- 2019Improved Generalization Bounds of Group Invariant / Equivariant Deep Networks via Quotient Feature Spacescited by 34
- 2020Scale Equivariance Improves Siamese Trackingcited by 32
- 2019Universal approximations of permutation invariant/equivariant functions by deep neural networkscited by 29
- 2020PDE-based Group Equivariant Convolutional Neural Networkscited by 28
- 2018Scale equivariance in CNNs with vector fieldscited by 20
- 2018A rotation-equivariant convolutional neural network model of primary visual cortexcited by 15
- 2017Balanced Excitation and Inhibition are Required for High-Capacity, Noise-Robust Neuronal Selectivitycited by 7
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 1.
2 papers.