# Where is the Information in a Deep Neural Network?

arXiv: https://arxiv.org/abs/1905.12213 · HTML: https://mlsanity.com/paper/1905.12213/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 30 of 42)

- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Fast Yet Effective Machine Unlearning](https://mlsanity.com/paper/2111.08947/) (2021)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Towards provably efficient quantum algorithms for large-scale machine-learning models](https://mlsanity.com/paper/2303.03428/) (2023)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [Taming AI Bots: Controllability of Neural States in Large Language Models](https://mlsanity.com/paper/2305.18449/) (2023)
- [Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively](https://mlsanity.com/paper/2211.01642/) (2022)
- [FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation](https://mlsanity.com/paper/2503.17940/) (2025)
- [Estimating informativeness of samples with Smooth Unique Information](https://mlsanity.com/paper/2101.06640/) (2021)
- [Does the Data Induce Capacity Control in Deep Learning?](https://mlsanity.com/paper/2110.14163/) (2021)
- [Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion](https://mlsanity.com/paper/2311.07682/) (2023)
- [Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States](https://mlsanity.com/paper/2211.10691/) (2022)
- [Towards Semantic Communication Protocols: A Probabilistic Logic Perspective](https://mlsanity.com/paper/2207.03920/) (2022)
- [AI Model Disgorgement: Methods and Choices](https://mlsanity.com/paper/2304.03545/) (2023)
- [AI Agents as Universal Task Solvers](https://mlsanity.com/paper/2510.12066/) (2025)
- [Neural Network Compression for Noisy Storage Devices](https://mlsanity.com/paper/2102.07725/) (2021)
- [On Leave-One-Out Conditional Mutual Information For Generalization](https://mlsanity.com/paper/2207.00581/) (2022)
- [Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behavior](https://mlsanity.com/paper/2511.02022/) (2025)
- [A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness](https://mlsanity.com/paper/2309.03004/) (2023)
- [Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint](https://mlsanity.com/paper/2509.06795/) (2025)
- [Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation](https://mlsanity.com/paper/2303.05161/) (2023)
- [Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AI](https://mlsanity.com/paper/2405.14061/) (2024)
- [Learning Capacity: A Measure of the Effective Dimensionality of a Model](https://mlsanity.com/paper/2305.17332/) (2023)
- [Quantifying and Localizing Usable Information Leakage from Neural Network Gradients](https://mlsanity.com/paper/2105.13929/) (2021)
- [Representation Learning with Conditional Information Flow Maximization](https://mlsanity.com/paper/2406.05510/) (2024)
- [Steering Information Utility in Key-Value Memory for Language Model Post-Training](https://mlsanity.com/paper/2507.05158/) (2025)
- [A Model of Understanding in Deep Learning Systems](https://mlsanity.com/paper/2604.04171/) (2026)
- [Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space](https://mlsanity.com/paper/2605.16600/) (2026)

## Abstract (arXiv)

Whatever information a deep neural network has gleaned from training data is encoded in its weights. How this information affects the response of the network to future data remains largely an open question. Indeed, even defining and measuring information entails some subtleties, since a trained network is a deterministic map, so standard information measures can be degenerate. We measure information in a neural network via the optimal trade-off between accuracy of the response and complexity of the weights, measured by their coding length. Depending on the choice of code, the definition can reduce to standard measures such as Shannon Mutual Information and Fisher Information. However, the more general definition allows us to relate information to generalization and invariance, through a novel notion of effective information in the activations of a deep network. We establish a novel relation between the information in the weights and the effective information in the activations, and use this result to show that models with low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs. These relations hinge not only on the architecture of the model, but also on how it is trained, highlighting the complex inter-dependency between the class of functions implemented by deep neural networks, the loss function used for training them from finite data, and the inductive bias implicit in the optimization.