mlsanityData to 7 Oct 2026

Provably Strict Generalisation Benefit for Invariance in Kernel Methods

arXiv 2106.02346 ↗First posted 4 Jun 2021stat.ML cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

It is a commonly held belief that enforcing invariance improves generalisation. Although this approach enjoys widespread popularity, it is only very recently that a rigorous theoretical demonstration of this benefit has been established. In this work we build on the function space perspective of Elesedy and Zaidi arXiv:2102.10333 to derive a strictly non-zero generalisation benefit of incorporating invariance in kernel ridge regression when the target is invariant to the action of a compact group. We study invariance enforced by feature averaging and find that generalisation is governed by a notion of effective dimension that arises from the interplay between the kernel and the group. In building towards this result, we find that the action of the group induces an orthogonal decomposition of both the reproducing kernel Hilbert space and its kernel, which may be of interest in its own right.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

22 of 49 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017Deep Setscited by 1,394
  3. 2016Group Equivariant Convolutional Networkscited by 1,154
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2018Spherical CNNscited by 452
  7. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  8. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  9. 2019On the Universality of Invariant Networkscited by 151
  10. 2017Equivariance Through Parameter-Sharingcited by 139
  11. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  12. 2018Universal approximations of invariant maps by neural networkscited by 127
10 more
  1. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 90
  2. 2021Learning with invariances in random features and kernel modelscited by 89
  3. 2020On the Benefits of Invariance in Neural Networkscited by 78
  4. 2016Generalization Error of Invariant Classifierscited by 42
  5. 20183D G-CNNs for Pulmonary Nodule Detectioncited by 36
  6. 2020Better, Faster Fermionic Neural Networkscited by 28
  7. 2020Improving Transformation Invariance in Contrastive Representation Learningcited by 16
  8. 2015Learning with Group Invariant Features: A Kernel Perspectivecited by 13
  9. 2016Local Group Invariant Representations via Orbit Embeddingscited by 12
  10. 2013Unsupervised Learning of Invariant Representations in Hierarchical Architecturescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 6; 2024: 6; 2025: 6; 2026: 4.

29 papers.

Sort:

  1. 2023Approximation-Generalization Trade-offs under (Approximate) Group Equivariancecited by 45
  2. 2024Equivariant Frames and the Impossibility of Continuous Canonicalizationcited by 41
  3. 2023The Exact Sample Complexity Gain from Invariances for Kernel Regressioncited by 27
  4. 2022A Theory of PAC Learnability under Transformation Invariancescited by 21
  5. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 17
  6. 2022Dimensionless machine learning: Imposing exact units equivariancecited by 16
  7. 2023Approximately Equivariant Graph Networkscited by 16
  8. 2022Machine learning and invariant theorycited by 13
  9. 2024On the hardness of learning under symmetriescited by 13
  10. 2021Partial Domain Adaptation without Domain Alignmentcited by 11
  11. 2023Towards fully covariant machine learningcited by 10
  12. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
17 more
  1. 2025Koopman-Equivariant Gaussian Processescited by 6
  2. 2025Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetrycited by 6
  3. 2025Learning with Exact Invariances in Polynomial Timecited by 5
  4. 2023Non-parametric Hypothesis Tests for Distributional Group Symmetrycited by 3
  5. 2024Tensor learning with orthogonal, Lorentz, and symplectic symmetriescited by 3
  6. 2026Efficient Learning and Symmetry Discovery under Exact Invariancescited by 2
  7. 2022Testing for Geometric Invariance and Equivariancecited by 1
  8. 2022A tradeoff between universality of equivariant models and learnability of symmetriescited by 1
  9. 2024Optimal Symmetries in Binary Classificationcited by 1
  10. 2025Statistical Learning Guarantees for Group-Invariant Barron Functionscited by 1
  11. 2023Returning The Favour: When Regression Benefits From Probabilistic Causal Knowledgecited by 0
  12. 2024Approximating invariant functions with the sorting trick is theoretically justifiedcited by 0
  13. 2025Generalization Bounds for Equivariant Networks on Markov Datacited by 0
  14. 2025Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbitscited by 0
  15. 2026The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalizationcited by 0
  16. 2026When and How to Canonize: A Generalization Perspectivecited by 0
  17. 2026Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimizationcited by 0