# Posterior Regularized Bayesian Neural Network Incorporating Soft and Hard Knowledge Constraints

arXiv: https://arxiv.org/abs/2210.08608 · HTML: https://mlsanity.com/paper/2210.08608/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Enhancing Explainability of Neural Networks through Architecture Constraints](https://mlsanity.com/paper/1901.03838/) (2019)
- [Bayesian Recurrent Neural Network Models for Forecasting and Quantifying Uncertainty in Spatial-Temporal Data](https://mlsanity.com/paper/1711.00636/) (2017)
- [Gaussian processes with linear operator inequality constraints](https://mlsanity.com/paper/1901.03134/) (2019)
- [Incorporating Interpretable Output Constraints in Bayesian Neural Networks](https://mlsanity.com/paper/2010.10969/) (2020)
- [A novel augmented Lagrangian method of multipliers for optimization with general inequality constraints](https://mlsanity.com/paper/2106.15044/) (2021)

## Built on by (most cited first, 3 of 3)

- [Bayesian Neural Networks with Domain Knowledge Priors](https://mlsanity.com/paper/2402.13410/) (2024)
- [Bayesian Entropy Neural Networks for Physics-Aware Prediction](https://mlsanity.com/paper/2407.01015/) (2024)
- [Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Bidding](https://mlsanity.com/paper/2412.19064/) (2024)

## Abstract (arXiv)

Neural Networks (NNs) have been widely {used in supervised learning} due to their ability to model complex nonlinear patterns, often presented in high-dimensional data such as images and text. However, traditional NNs often lack the ability for uncertainty quantification. Bayesian NNs (BNNS) could help measure the uncertainty by considering the distributions of the NN model parameters. Besides, domain knowledge is commonly available and could improve the performance of BNNs if it can be appropriately incorporated. In this work, we propose a novel Posterior-Regularized Bayesian Neural Network (PR-BNN) model by incorporating different types of knowledge constraints, such as the soft and hard constraints, as a posterior regularization term. Furthermore, we propose to combine the augmented Lagrangian method and the existing BNN solvers for efficient inference. The experiments in simulation and two case studies about aviation landing prediction and solar energy output prediction have shown the knowledge constraints and the performance improvement of the proposed model over traditional BNNs without the constraints.