mlsanityData to 7 Oct 2026

Universal approximation and model compression for radial neural networks

arXiv 2107.02550 ↗First posted 6 Jul 2021cs.LG math.RTcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by a function depending only on their norm. We call such networks radial neural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality. We prove universal approximation theorems for radial neural networks, including in the more difficult cases of bounded widths and unbounded domains. Our proof techniques are novel, distinct from those in the pointwise case. Additionally, radial neural networks exhibit a rich group of orthogonal change-of-basis symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to a practical lossless model compression algorithm. Optimization of the compressed model by gradient descent is equivalent to projected gradient descent for the full model.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 50 of its 70 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

50 of 70 references.

References · 50

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2016Group Equivariant Convolutional Networkscited by 1,154
  7. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  8. 2017Dynamic Routing Between Capsulescited by 689
  9. 2017Self-Normalizing Neural Networkscited by 591
  10. 2020What is the State of Neural Network Pruning?cited by 578
  11. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  12. 2013Do Deep Nets Really Need to be Deep?cited by 466
38 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  2. 2019General E(2)-Equivariant Steerable CNNscited by 441
  3. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  4. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  5. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  6. 2016Steerable CNNscited by 298
  7. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  8. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  9. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  10. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 262
  11. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  12. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  13. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  14. 2019Universal Approximation with Deep Narrow Networkscited by 198
  15. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 193
  16. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  17. 2017Equivariance Through Parameter-Sharingcited by 139
  18. 2018Universal approximations of invariant maps by neural networkscited by 127
  19. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  20. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  21. 2015Convolutional neural networks with low-rank regularizationcited by 102
  22. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  23. 2016Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classificationcited by 93
  24. 2016Exploiting Cyclic Symmetry in Convolutional Neural Networkscited by 91
  25. 2015Neural Network with Unbounded Activation Functions is Universal Approximatorcited by 63
  26. 2020Universal Equivariant Multilayer Perceptronscited by 49
  27. 2020A Wigner-Eckart Theorem for Group Equivariant Convolution Kernelscited by 46
  28. 2020Lossless Compression of Deep Neural Networkscited by 34
  29. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 26
  30. 2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
  31. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  32. 2020The Representation Theory of Neural Networkscited by 18
  33. 2020Neural Teleportationcited by 17
  34. 2018Rotation Equivariance and Invariance in Convolutional Neural Networkscited by 12
  35. 2020Lossless Compression of Structured Convolutional Models via Liftingcited by 11
  36. 2019Approximation capabilities of neural networks on unbounded domainscited by 7
  37. 2021Size and Depth Separation in Approximating Benign Functions with Neural Networkscited by 7
  38. 2015An exploration of parameter redundancy in deep networks with circulant projectionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

7 papers.

  1. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  4. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  5. 2022Quiver neural networkscited by 4
  6. 2025Teleportation With Null Space Gradient Projection for Optimization Accelerationcited by 2
  7. 2026Discovering Symmetries in Neural Network Parameter Spacescited by 0