mlsanityData to 14 Oct 2026

Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks

arXiv 2312.08550 ↗First posted 13 Dec 2023cs.LG cs.AI eess.SPcited by 23

Not on the map. Why?

Abstract (arXiv)

In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 18 of its 59 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

18 of 59 references.

References · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  3. 2016Group Equivariant Convolutional Networkscited by 1,165
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  5. 2017Dynamic Routing Between Capsulescited by 689
  6. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  7. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  8. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  9. 2017Deep Complex Networkscited by 217
  10. 2022Relative representations enable zero-shot latent space communicationcited by 191
  11. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  12. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
6 more
  1. 2021A Survey of Complex-Valued Neural Networkscited by 61
  2. 2018Emergence of grid-like representations by training recurrent neural networks to perform spatial localizationcited by 52
  3. 2022Complex-Valued Autoencoders for Object Discoverycited by 40
  4. 2022Bispectral Neural Networkscited by 18
  5. 2022Actionable Neural Representations: Grid Cells from Minimal Constraintscited by 10
  6. 2023Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cellscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
  3. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  4. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  5. 2024On the Geometry and Optimization of Polynomial Convolutional Networkscited by 11
  6. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 10
  7. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  8. 2025Flopping for FLOPs: Leveraging equivariance for computational efficiencycited by 8
  9. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  10. 2025Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spacescited by 6
  11. 2025Representation biases: will we achieve complete understanding by analyzing representations?cited by 6
  12. 2024Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensorscited by 5
11 more
  1. 2024Relative Representations: Topological and Geometric Perspectivescited by 4
  2. 2024On the universality of neural encodings in CNNscited by 4
  3. 2024Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicinecited by 2
  4. 2025On the Complexity-Faithfulness Trade-off of Gradient-Based Explanationscited by 2
  5. 2026Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spacescited by 1
  6. 2025Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Additioncited by 0
  7. 2025The Role of Feature Interactions in Graph-based Tabular Deep Learningcited by 0
  8. 2025Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbitscited by 0
  9. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  10. 2026Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attentioncited by 0
  11. 2026Math for AI safety: an invitation for mathematicianscited by 0