mlsanityData to 7 Oct 2026

Neuronal diversity can improve machine learning for physics and beyond

arXiv 2204.04348 ↗First posted 9 Apr 2022cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses. Examples include conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning H'enon-Heiles stellar orbits and the swing of a video recorded pendulum clock. Such textitlearned diversity provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artificial systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 75 references that are in the map. First posted on arXiv 9 Apr 2022.

Builds on

25 of 75 references.

References · 25

  1. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  2. 2019Hamiltonian Neural Networkscited by 702
  3. 2013Network In Networkcited by 600
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  7. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 143
  10. 2019Hamiltonian Generative Networkscited by 136
  11. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  12. 2020A survey on modern trainable activation functionscited by 89
13 more
  1. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  2. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  3. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  4. 2015Diversity Networks: Neural Network Compression Using Determinantal Point Processescited by 25
  5. 2020Scaling Down Deep Learning with MNIST-1Dcited by 23
  6. 2021Adaptable Hamiltonian neural networkscited by 21
  7. 2018Learning Combinations of Activation Functionscited by 20
  8. 2020Forecasting Hamiltonian dynamics without canonical coordinatescited by 17
  9. 2017On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbationscited by 14
  10. 2018Toward an AI Physicist for Unsupervised Learningcited by 12
  11. 2017Stochastic Gradient Descent in Continuous Time: A Central Limit Theoremcited by 10
  12. 2014Frequently Asked Questions for: The Atoms of Neural Computationcited by 4
  13. 2020Mastering high-dimensional dynamics with Hamiltonian neural networkscited by 3

Built on by

Not cited yet.