# Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Concepts

arXiv: https://arxiv.org/abs/2302.13095 · HTML: https://mlsanity.com/paper/2302.13095/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 25 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation](https://mlsanity.com/paper/1903.10992/) (2019)
- [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](https://mlsanity.com/paper/2002.04138/) (2020)
- [On the Connection Between Adversarial Robustness and Saliency Map Interpretability](https://mlsanity.com/paper/1905.04172/) (2019)
- [A Unified Approach to Interpreting and Boosting Adversarial Transferability](https://mlsanity.com/paper/2010.04055/) (2020)
- [Robustness of Bayesian Neural Networks to Gradient-Based Attacks](https://mlsanity.com/paper/2002.04359/) (2020)
- [Discovering and Explaining the Representation Bottleneck of DNNs](https://mlsanity.com/paper/2111.06236/) (2021)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Interpreting and Boosting Dropout from a Game-Theoretic View](https://mlsanity.com/paper/2009.11729/) (2020)
- [Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes](https://mlsanity.com/paper/1906.05323/) (2019)
- [A Unified Game-Theoretic Interpretation of Adversarial Robustness](https://mlsanity.com/paper/2111.03536/) (2021)
- [Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?](https://mlsanity.com/paper/2105.10719/) (2021)
- [Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks](https://mlsanity.com/paper/1806.00667/) (2018)
- [Dropout as a Regularizer of Interaction Effects](https://mlsanity.com/paper/2007.00823/) (2020)

## Built on by (most cited first, 9 of 9)

- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Technical Report: Quantifying and Analyzing the Generalization Power of a DNN](https://mlsanity.com/paper/2505.06993/) (2025)
- [Why pre-training is beneficial for downstream classification tasks?](https://mlsanity.com/paper/2410.08455/) (2024)
- [Cross-LLM Consistency in Inference: Evidence from Shared Interactions](https://mlsanity.com/paper/2606.08129/) (2026)
- [Disentangling Regional Primitives for Image Generation](https://mlsanity.com/paper/2410.04421/) (2024)
- [S2CFormer: Revisiting the RD-Latency Trade-off in Transformer-based Learned Image Compression](https://mlsanity.com/paper/2502.00700/) (2025)
- [Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective](https://mlsanity.com/paper/2605.17967/) (2026)
- [A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions](https://mlsanity.com/paper/2607.08776/) (2026)
- [Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions](https://mlsanity.com/paper/2608.18539/) (2026)

## Abstract (arXiv)

In this paper, we focus on mean-field variational Bayesian Neural Networks (BNNs) and explore the representation capacity of such BNNs by investigating which types of concepts are less likely to be encoded by the BNN. It has been observed and studied that a relatively small set of interactive concepts usually emerge in the knowledge representation of a sufficiently-trained neural network, and such concepts can faithfully explain the network output. Based on this, our study proves that compared to standard deep neural networks (DNNs), it is less likely for BNNs to encode complex concepts. Experiments verify our theoretical proofs. Note that the tendency to encode less complex concepts does not necessarily imply weak representation power, considering that complex concepts exhibit low generalization power and high adversarial vulnerability. The code is available at https://github.com/sjtu-xai-lab/BNN-concepts.