# Belief Propagation in Conditional RBMs for Structured Prediction

arXiv: https://arxiv.org/abs/1703.00986 · HTML: https://mlsanity.com/paper/1703.00986/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2017.

## Built on by (most cited first, 4 of 4)

- [Belief Propagation as Diffusion](https://mlsanity.com/paper/2107.12230/) (2021)
- [Understanding the Behavior of Belief Propagation](https://mlsanity.com/paper/2209.05464/) (2022)
- [Local Max-Entropy and Free Energy Principles, Belief Diffusions and their Singularities](https://mlsanity.com/paper/2310.02946/) (2023)
- [Topology-Aware Exploration of Energy-Based Models Equilibrium: Toric QC-LDPC Codes and Hyperbolic MET QC-LDPC Codes](https://mlsanity.com/paper/2401.14749/) (2024)

## Abstract (arXiv)

Restricted Boltzmann machines~(RBMs) and conditional RBMs~(CRBMs) are popular models for a wide range of applications. In previous work, learning on such models has been dominated by contrastive divergence~(CD) and its variants. Belief propagation~(BP) algorithms are believed to be slow for structured prediction on conditional RBMs~(e.g., Mnih et al. [2011]), and not as good as CD when applied in learning~(e.g., Larochelle et al. [2012]). In this work, we present a matrix-based implementation of belief propagation algorithms on CRBMs, which is easily scalable to tens of thousands of visible and hidden units. We demonstrate that, in both maximum likelihood and max-margin learning, training conditional RBMs with BP as the inference routine can provide significantly better results than current state-of-the-art CD methods on structured prediction problems. We also include practical guidelines on training CRBMs with BP, and some insights on the interaction of learning and inference algorithms for CRBMs.