# Natural-Parameter Networks: A Class of Probabilistic Neural Networks

arXiv: https://arxiv.org/abs/1611.00448 · HTML: https://mlsanity.com/paper/1611.00448/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2016.

## Built on by (most cited first, 30 of 34)

- [PreDiff: Precipitation Nowcasting with Latent Diffusion Models](https://mlsanity.com/paper/2307.10422/) (2023)
- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [Quantifying Uncertainty in Deep Spatiotemporal Forecasting](https://mlsanity.com/paper/2105.11982/) (2021)
- [TacticZero: Learning to Prove Theorems from Scratch with Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09756/) (2021)
- [TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning](https://mlsanity.com/paper/2505.11737/) (2025)
- [On Calibration of LLM-based Guard Models for Reliable Content Moderation](https://mlsanity.com/paper/2410.10414/) (2024)
- [Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty](https://mlsanity.com/paper/2104.12446/) (2021)
- [Propagating State Uncertainty Through Trajectory Forecasting](https://mlsanity.com/paper/2110.03267/) (2021)
- [ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference](https://mlsanity.com/paper/2211.11435/) (2022)
- [Blurs Behave Like Ensembles: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness](https://mlsanity.com/paper/2105.12639/) (2021)
- [Deep Probability Estimation](https://mlsanity.com/paper/2111.10734/) (2021)
- [Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions](https://mlsanity.com/paper/2105.04471/) (2021)
- [FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?](https://mlsanity.com/paper/2205.11656/) (2022)
- [Training-Free Bayesianization for Low-Rank Adapters of Large Language Models](https://mlsanity.com/paper/2412.05723/) (2024)
- [Uncertainty Estimation for Molecules: Desiderata and Methods](https://mlsanity.com/paper/2306.14916/) (2023)
- [Uncertainty Quantification via Stable Distribution Propagation](https://mlsanity.com/paper/2402.08324/) (2024)
- [Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models](https://mlsanity.com/paper/2406.12649/) (2024)
- [GOSH: Task Scheduling Using Deep Surrogate Models in Fog Computing Environments](https://mlsanity.com/paper/2112.08916/) (2021)
- [CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework](https://mlsanity.com/paper/2212.03965/) (2022)
- [Delving into Differentially Private Transformer](https://mlsanity.com/paper/2405.18194/) (2024)
- [Context Uncertainty in Contextual Bandits with Applications to Recommender Systems](https://mlsanity.com/paper/2202.00805/) (2022)
- [Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise](https://mlsanity.com/paper/2103.08497/) (2021)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [DPFormer: Learning Differentially Private Transformer on Long-Tailed Data](https://mlsanity.com/paper/2305.17633/) (2023)
- [Learning by Erasing: Conditional Entropy based Transferable Out-Of-Distribution Detection](https://mlsanity.com/paper/2204.11041/) (2022)
- [Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing](https://mlsanity.com/paper/2306.06599/) (2023)
- [Bootstrap Your Own Variance](https://mlsanity.com/paper/2312.03213/) (2023)
- [Towards Domain Adaptive Neural Contextual Bandits](https://mlsanity.com/paper/2406.09564/) (2024)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)
- [Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2505.11731/) (2025)

## Abstract (arXiv)

Neural networks (NN) have achieved state-of-the-art performance in various applications. Unfortunately in applications where training data is insufficient, they are often prone to overfitting. One effective way to alleviate this problem is to exploit the Bayesian approach by using Bayesian neural networks (BNN). Another shortcoming of NN is the lack of flexibility to customize different distributions for the weights and neurons according to the data, as is often done in probabilistic graphical models. To address these problems, we propose a class of probabilistic neural networks, dubbed natural-parameter networks (NPN), as a novel and lightweight Bayesian treatment of NN. NPN allows the usage of arbitrary exponential-family distributions to model the weights and neurons. Different from traditional NN and BNN, NPN takes distributions as input and goes through layers of transformation before producing distributions to match the target output distributions. As a Bayesian treatment, efficient backpropagation (BP) is performed to learn the natural parameters for the distributions over both the weights and neurons. The output distributions of each layer, as byproducts, may be used as second-order representations for the associated tasks such as link prediction. Experiments on real-world datasets show that NPN can achieve state-of-the-art performance.