# Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation

arXiv: https://arxiv.org/abs/2303.09154 · HTML: https://mlsanity.com/paper/2303.09154/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 70 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Promises and Pitfalls of Black-Box Concept Learning Models](https://mlsanity.com/paper/2106.13314/) (2021)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Do Concept Bottleneck Models Learn as Intended?](https://mlsanity.com/paper/2105.04289/) (2021)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Explainable Object-induced Action Decision for Autonomous Vehicles](https://mlsanity.com/paper/2003.09405/) (2020)
- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [Credal Bayesian Deep Learning](https://mlsanity.com/paper/2302.09656/) (2023)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [A Bayesian information criterion for singular models](https://mlsanity.com/paper/1309.0911/) (2013)
- [The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation](https://mlsanity.com/paper/2008.01304/) (2020)
- [Marginal likelihood and model selection for Gaussian latent tree and forest models](https://mlsanity.com/paper/1412.8285/) (2014)
- [Upper Bound of Bayesian Generalization Error in Non-negative Matrix Factorization](https://mlsanity.com/paper/1612.04112/) (2016)
- [Estimating Real Log Canonical Thresholds](https://mlsanity.com/paper/1906.01341/) (2019)
- [Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures](https://mlsanity.com/paper/2203.06884/) (2022)
- [Learning to Predict with Supporting Evidence: Applications to Clinical Risk Prediction](https://mlsanity.com/paper/2103.02768/) (2021)
- [Towards learning to explain with concept bottleneck models: mitigating information leakage](https://mlsanity.com/paper/2211.03656/) (2022)
- [Variational Approximation Error in Bayesian Non-negative Matrix Factorization](https://mlsanity.com/paper/1809.02963/) (2018)
- [Mathematical Theory of Bayesian Statistics for Unknown Information Source](https://mlsanity.com/paper/2206.05630/) (2022)
- [Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization](https://mlsanity.com/paper/1709.04212/) (2017)

## Built on by (most cited first, 3 of 3)

- [Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM](https://mlsanity.com/paper/2403.09206/) (2024)
- [Scaling Laws for Uncertainty in Deep Learning](https://mlsanity.com/paper/2506.09648/) (2025)
- [An Asymptotic Equation Linking WAIC and WBIC in Singular Models](https://mlsanity.com/paper/2505.13902/) (2025)

## Abstract (arXiv)

Concept bottleneck model (CBM) is a ubiquitous method that can interpret neural networks using concepts. In CBM, concepts are inserted between the output layer and the last intermediate layer as observable values. This helps in understanding the reason behind the outputs generated by the neural networks: the weights corresponding to the concepts from the last hidden layer to the output layer. However, it has not yet been possible to understand the behavior of the generalization error in CBM since a neural network is a singular statistical model in general. When the model is singular, a one to one map from the parameters to probability distributions cannot be created. This non-identifiability makes it difficult to analyze the generalization performance. In this study, we mathematically clarify the Bayesian generalization error and free energy of CBM when its architecture is three-layered linear neural networks. We also consider a multitask problem where the neural network outputs not only the original output but also the concepts. The results show that CBM drastically changes the behavior of the parameter region and the Bayesian generalization error in three-layered linear neural networks as compared with the standard version, whereas the multitask formulation does not.