mlsanityData to 14 Oct 2026

Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines

arXiv 2605.19178 ↗First posted 18 May 2026cond-mat.dis-nn cond-mat.stat-mech cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The great success of neural networks primarily arises from the presence of the large number of weight parameters combined with nonlinearities in the input-output relationship of single neurons. In this work, we study the relationship between the statistical properties of the weights and the nonlinearity of the hidden unit in Restricted Boltzmann Machines (RBMs) on the one side, and the distribution induced on binary visible units. We do this for four commonly used activation functions: Linear, Step, ReLU, and Exponential, and make qualitative predictions about the ability of these models to learn distributions with strong higher order interactions over the visible nodes. We show that in general, in an ensemble of RBMs with Gaussian weights, these distributions are rare and hard to learn, except when the hidden unit activation function is an Exponential.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

25 of 47 references.

References · 25

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  5. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  10. 2020Restricted Boltzmann Machine, recent advances and mean-field theorycited by 31
  11. 2017Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priorscited by 25
  12. 2022On the Generalization Mystery in Deep Learningcited by 25
13 more
  1. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 24
  2. 2016Emergence of Compositional Representations in Restricted Boltzmann Machinescited by 23
  3. 2024Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networkscited by 23
  4. 2016Phase transitions in Restricted Boltzmann Machines with generic priorscited by 20
  5. 2019Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activationcited by 14
  6. 2021Restricted Boltzmann Machines as Models of Interacting Variablescited by 9
  7. 2025Inferring Higher-Order Couplings with Neural Networkscited by 7
  8. 2020Efficiency of local learning rules in threshold-linear associative networkscited by 6
  9. 2025The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applicationscited by 4
  10. 2024Solution space and storage capacity of fully connected two-layer neural networks with generic activation functionscited by 3
  11. 2024Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machinescited by 3
  12. 2024The effect of priors on Learning with Restricted Boltzmann Machinescited by 3
  13. 2025On the role of non-linear latent features in bipartite generative neural networkscited by 1

Built on by

Not cited yet.