# A topological insight into restricted Boltzmann machines

arXiv: https://arxiv.org/abs/1604.05978 · HTML: https://mlsanity.com/paper/1604.05978/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2016.

## Built on by (most cited first, 30 of 32)

- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity](https://mlsanity.com/paper/2106.14568/) (2021)
- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers](https://mlsanity.com/paper/2302.02596/) (2023)
- [Dynamic Sparse Network for Time Series Classification: Learning What to "see''](https://mlsanity.com/paper/2212.09840/) (2022)
- [Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training](https://mlsanity.com/paper/2306.12230/) (2023)
- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2312.04727/) (2023)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [Sparse Training Theory for Scalable and Efficient Agents](https://mlsanity.com/paper/2103.01636/) (2021)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models](https://mlsanity.com/paper/2505.01731/) (2025)
- [Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance](https://mlsanity.com/paper/2203.02770/) (2022)
- [Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks](https://mlsanity.com/paper/2303.07200/) (2023)
- [Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training](https://mlsanity.com/paper/2205.15322/) (2022)
- [Pruning Before Training May Improve Generalization, Provably](https://mlsanity.com/paper/2301.00335/) (2023)
- [Self-Regulated Neurogenesis for Online Data-Incremental Learning](https://mlsanity.com/paper/2403.14684/) (2024)
- [Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability](https://mlsanity.com/paper/2306.00045/) (2023)
- [REST: Enhancing Group Robustness in DNNs through Reweighted Sparse Training](https://mlsanity.com/paper/2312.03044/) (2023)
- [Nerva: a Truly Sparse Implementation of Neural Networks](https://mlsanity.com/paper/2407.17437/) (2024)
- [Self-building Neural Networks](https://mlsanity.com/paper/2304.01086/) (2023)
- [Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers](https://mlsanity.com/paper/2305.18382/) (2023)
- [Are Sparse Neural Networks Better Hard Sample Learners?](https://mlsanity.com/paper/2409.09196/) (2024)
- [Sparse-to-Sparse Training of Diffusion Models](https://mlsanity.com/paper/2504.21380/) (2025)
- [Evolving Neural Networks with Optimal Balance between Information Flow and Connections Cost](https://mlsanity.com/paper/2202.06163/) (2022)
- [Graph-based methods coupled with specific distributional distances for adversarial attack detection](https://mlsanity.com/paper/2306.00042/) (2023)

## Abstract (arXiv)

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.