mlsanityData to 7 Oct 2026

Equivariant Neural Tangent Kernels

arXiv 2406.06504 ↗First posted 10 Jun 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Little is known about the training dynamics of equivariant neural networks, in particular how it compares to data augmented training of their non-equivariant counterparts. Recently, neural tangent kernels (NTKs) have emerged as a powerful tool to analytically study the training dynamics of wide neural networks. In this work, we take an important step towards a theoretical understanding of training dynamics of equivariant models by deriving neural tangent kernels for a broad class of equivariant architectures based on group convolutions. As a demonstration of the capabilities of our framework, we show an interesting relationship between data augmentation and group convolutional networks. Specifically, we prove that they share the same expected prediction at all training times and even off-manifold. In this sense, they have the same training dynamics. We demonstrate in numerical experiments that this still holds approximately for finite-width ensembles. By implementing equivariant NTKs for roto-translations in the plane (G=Cnltimesℝ2) and 3d rotations (G=SO(3)), we show that equivariant NTKs outperform their non-equivariant counterparts as kernel predictors for histological image classification and quantum mechanical property prediction.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 53 of its 73 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

53 of 73 references.

References · 53

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Group Equivariant Convolutional Networkscited by 1,154
  4. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  5. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  6. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  7. 2017Deep Neural Networks as Gaussian Processescited by 617
  8. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2018Spherical CNNscited by 452
  11. 2019General E(2)-Equivariant Steerable CNNscited by 441
  12. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
38 more
  1. 2016Steerable CNNscited by 298
  2. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  3. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  4. 2023Spherical Fourier Neural Operators: Learning Stable Dynamics on the Spherecited by 225
  5. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 193
  6. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  7. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  8. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  9. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  10. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  11. 2020Lorentz Group Equivariant Neural Network for Particle Physicscited by 120
  12. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  13. 2018Roto-Translation Covariant Convolutional Networks for Medical Image Analysiscited by 91
  14. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  15. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  16. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  17. 2021SE(3)-equivariant prediction of molecular wavefunctions and electronic densitiescited by 67
  18. 2023A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systemscited by 65
  19. 2021Geometric Deep Learning and Equivariant Neural Networkscited by 64
  20. 2018DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applicationscited by 61
  21. 2020Neural Networks and Quantum Field Theorycited by 58
  22. 2023Swallowing the Bitter Pill: Simplified Scalable Conformer Generationcited by 55
  23. 2021Equivariant neural networks for inverse problemscited by 29
  24. 2022Diffeomorphic Counterfactuals with Generative Modelscited by 29
  25. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  26. 2022Equivariance versus Augmentation for Spherical Imagescited by 24
  27. 2022Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Modelscited by 22
  28. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  29. 2023Scaling Spherical CNNscited by 21
  30. 2021Rotation-Equivariant Deep Learning for Diffusion MRIcited by 20
  31. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  32. 2024Emergent Equivariance in Deep Ensemblescited by 18
  33. 2021Equivariant Filters for Efficient Tracking in 3D Imagingcited by 17
  34. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  35. 2023Structures of Neural Network Effective Theoriescited by 13
  36. 2023Sample based Explanations via Generalized Representerscited by 10
  37. 2022On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networkscited by 9
  38. 2023Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Modelscited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  2. 2025Finite-Width Neural Tangent Kernels from Feynman Diagramscited by 3
  3. 2025Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbitscited by 0
  4. 2026Boosting Data Augmentation with Stochastic Weight Averagingcited by 0