# Generalization Bounds for Neural Belief Propagation Decoders

arXiv: https://arxiv.org/abs/2305.10540 · HTML: https://mlsanity.com/paper/2305.10540/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 30 of 31)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Generalization and Representational Limits of Graph Neural Networks](https://mlsanity.com/paper/2002.06157/) (2020)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks](https://mlsanity.com/paper/2012.07690/) (2020)
- [Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors](https://mlsanity.com/paper/1712.09376/) (2017)
- [In Search of Robust Measures of Generalization](https://mlsanity.com/paper/2010.11924/) (2020)
- [Deep Learning-Based Channel Estimation](https://mlsanity.com/paper/1810.05893/) (2018)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [A Theory of Universal Learning](https://mlsanity.com/paper/2011.04483/) (2020)
- [Neural Enhanced Belief Propagation on Factor Graphs](https://mlsanity.com/paper/2003.01998/) (2020)
- [Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels](https://mlsanity.com/paper/1911.03038/) (2019)
- [Deepcode: Feedback Codes via Deep Learning](https://mlsanity.com/paper/1807.00801/) (2018)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [On Generalization Bounds of a Family of Recurrent Neural Networks](https://mlsanity.com/paper/1910.12947/) (2019)
- [Deep Learning for Wireless Communications](https://mlsanity.com/paper/2005.06068/) (2020)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [Non-Vacuous Generalisation Bounds for Shallow Neural Networks](https://mlsanity.com/paper/2202.01627/) (2022)
- [Deep Learning for Wireless Communications: An Emerging Interdisciplinary Paradigm](https://mlsanity.com/paper/2007.05952/) (2020)
- [Deep Neural Network Architectures for Modulation Classification](https://mlsanity.com/paper/1712.00443/) (2017)
- [Deep Learning for Direct Hybrid Precoding in Millimeter Wave Massive MIMO Systems](https://mlsanity.com/paper/1905.13212/) (2019)

## Built on by (most cited first, 4 of 4)

- [Generalization Bounds for Transformer Channel Decoders](https://mlsanity.com/paper/2601.06969/) (2026)
- [Online Optimization for Learning to Communicate over Time-Correlated Channels](https://mlsanity.com/paper/2409.00575/) (2024)
- [Action-List Reinforcement Learning Syndrome Decoding for Binary Linear Block Codes](https://mlsanity.com/paper/2507.17893/) (2025)
- [Data-Driven Deep MIMO Detection:Network Architectures and Generalization Analysis](https://mlsanity.com/paper/2602.20178/) (2026)

## Abstract (arXiv)

Machine learning based approaches are being increasingly used for designing decoders for next generation communication systems. One widely used framework is neural belief propagation (NBP), which unfolds the belief propagation (BP) iterations into a deep neural network and the parameters are trained in a data-driven manner. NBP decoders have been shown to improve upon classical decoding algorithms. In this paper, we investigate the generalization capabilities of NBP decoders. Specifically, the generalization gap of a decoder is the difference between empirical and expected bit-error-rate(s). We present new theoretical results which bound this gap and show the dependence on the decoder complexity, in terms of code parameters (blocklength, message length, variable/check node degrees), decoding iterations, and the training dataset size. Results are presented for both regular and irregular parity-check matrices. To the best of our knowledge, this is the first set of theoretical results on generalization performance of neural network based decoders. We present experimental results to show the dependence of generalization gap on the training dataset size, and decoding iterations for different codes.