mlsanityData to 7 Oct 2026

Deep learning via message passing algorithms based on belief propagation

arXiv 2110.14583 ↗First posted 27 Oct 2021cs.LG cond-mat.dis-nn stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Message-passing algorithms based on the Belief Propagation (BP) equations constitute a well-known distributed computational scheme. It is exact on tree-like graphical models and has also proven to be effective in many problems defined on graphs with loops (from inference to optimization, from signal processing to clustering). The BP-based scheme is fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present and adapt to mini-batch training on GPUs a family of BP-based message-passing algorithms with a reinforcement field that biases distributions towards locally entropic solutions. These algorithms are capable of training multi-layer neural networks with discrete weights and activations with performance comparable to SGD-inspired heuristics (BinaryNet) and are naturally well-adapted to continual learning. Furthermore, using these algorithms to estimate the marginals of the weights allows us to make approximate Bayesian predictions that have higher accuracy than point-wise solutions.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

32 of 67 references.

References · 32

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  8. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  9. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  10. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  11. 2013Expectation Propagation for approximate Bayesian inferencecited by 172
  12. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
20 more
  1. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  2. 2020Neural Enhanced Belief Propagation on Factor Graphscited by 55
  3. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  4. 2018Entropy and mutual information in models of deep neural networkscited by 50
  5. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  6. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  7. 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
  8. 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
  9. 2020Belief Propagation Neural Networkscited by 31
  10. 2021Synaptic metaplasticity in binarized neural networkscited by 28
  11. 2020Entropic gradient descent algorithms and wide flat minimacited by 25
  12. 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
  13. 2019The spiked matrix model with generative priorscited by 20
  14. 2019Mean-field inference methods for neural networkscited by 16
  15. 2017Multi-Layer Generalized Linear Estimationcited by 12
  16. 2017Inference in Deep Networks in High Dimensionscited by 12
  17. 2020Belief Propagation Reloaded: Learning BP-Layers for Labeling Problemscited by 9
  18. 2016Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networkscited by 6
  19. 2016Learning may need only a few bits of synaptic precisioncited by 5
  20. 2017Streaming Bayesian inference: theoretical limits and mini-batch approximate message-passingcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 3; 2025: 2.

12 papers.

  1. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9
  2. 2023Complex Recurrent Spectral Networkcited by 5
  3. 2024FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustnesscited by 5
  4. 2023Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problemscited by 3
  5. 2023The Decimation Scheme for Symmetric Matrix Factorizationcited by 2
  6. 2024Bayesian Deep Learning Via Expectation Maximization and Turbo Deep Approximate Message Passingcited by 2
  7. 2025Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Searchcited by 2
  8. 2023Exact Fractional Inference via Re-Parametrization & Interpolation between Tree-Re-Weighted- and Belief Propagation- Algorithmscited by 1
  9. 2023Learning in Deep Factor Graphs with Gaussian Belief Propagationcited by 1
  10. 2024Training Multi-Layer Binary Neural Networks With Local Binary Error Signalscited by 1
  11. 2023Engineered Ordinary Differential Equations as Classification Algorithm (EODECA): thorough characterization and testingcited by 0
  12. 2025BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Trainingcited by 0