mlsanityData to 14 Oct 2026

Information Plane Analysis of Binary Neural Networks

arXiv 2605.03636 ↗First posted 5 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Information plane (IP) analysis has been suggested to study the training dynamics of deep neural networks through mutual information (MI) between inputs, representations, and targets. However, its statistical validity is often compromised by the difficulty of estimating MI from samples of high-dimensional, deterministic representations. In this work, we perform IP analyses on binary neural networks (BNNs) where activations are discrete and MI is finite. We characterise the finite-sample behaviour of the plug-in entropy estimator and identify regimes for sample size N and representation dimensionality D under which MI estimates are reliable. Outside these regimes, we show that empirical MI estimates saturate to log2 N, rendering IP trajectories uninformative. Restricting attention to the reliable regime, we train 375 BNNs to investigate the existence of late-stage compression phases and the relationship between compressed representations and generalisation performance. Our results show that while late-stage compression is frequently observed, compressed latent representations do not consistently correlate with improved generalization performance. Instead, the relationship between compression and generalisation is highly dependent on task, architecture, and regularisation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 15 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

10 of 15 references.

References · 10

  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  4. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  5. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  6. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  7. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 37
  8. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 15
  9. 2017Layer-wise Learning of Stochastic Neural Networks with Information Bottleneckcited by 4
  10. 2020Understanding Learning Dynamics of Binary Neural Networks via Information Bottleneckcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
  2. 2025Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis
  3. 2022A Measure of the Complexity of Neural Representations based on Partial Information Decomposition
  4. 2024Collective variables of neural networks: empirical time evolution and scaling laws
  5. 2025A Generalized Information Bottleneck Theory of Deep Learning
  6. 2026Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis
  7. 2022IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networks
  8. 2026Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective