mlsanityData to 14 Oct 2026

Cauchy-Schwarz Divergence Information Bottleneck for Regression

arXiv 2404.17951 ↗First posted 27 Apr 2024cs.LG cs.IT math.ITcited by 9

Not on the map. Why?

Abstract (arXiv)

The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation t by striking a trade-off between a compression term I(x;t) and a prediction term I(y;t), where I(·;·) refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at urlhttps://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 41 of its 71 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

41 of 71 references.

References · 41

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2016Deep Variational Information Bottleneckcited by 930
  5. 2018MINE: Mutual Information Neural Estimationcited by 683
  6. 2019On Variational Bounds of Mutual Informationcited by 559
  7. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  8. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  9. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  10. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  11. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  12. 2020A Theory of Usable Information Under Computational Constraintscited by 162
29 more
  1. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  3. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  4. 2020The Conditional Entropy Bottleneckcited by 75
  5. 2017Nonlinear Information Bottleneckcited by 71
  6. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  7. 2016The deterministic information bottleneckcited by 42
  8. 2017Estimating Mixture Entropy with Pairwise Distancescited by 36
  9. 2019Explaining a black-box using Deep Variational Information Bottleneck Approachcited by 36
  10. 2019Robust Learning with the Hilbert-Schmidt Independence Criterioncited by 36
  11. 2021KL Guided Domain Adaptationcited by 35
  12. 2018Caveats for information bottleneck in deterministic scenarioscited by 34
  13. 2020Learning Optimal Representations with the Decodable Information Bottleneckcited by 33
  14. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 32
  15. 2021PAC-Bayes Information Bottleneckcited by 28
  16. 2016Conditional Generative Moment-Matching Networkscited by 24
  17. 2020Disentangled Information Bottleneckcited by 24
  18. 2021Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustnesscited by 20
  19. 2019The Convex Information Bottleneck Lagrangiancited by 16
  20. 2021Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Explorationcited by 15
  21. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 15
  22. 2018Adversarial Attacks, Regression, and Numerical Stability Regularizationcited by 14
  23. 2019Adaptive Estimators Show Information Compression in Deep Neural Networkscited by 14
  24. 2021Cauchy-Schwarz Regularized Autoencodercited by 13
  25. 2022Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimalitycited by 9
  26. 2023The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Makingcited by 6
  27. 2022Bounding generalization error with input compression: An empirical study with infinite-width networkscited by 4
  28. 2020The Role of Mutual Information in Variational Classifierscited by 3
  29. 2023Generalized Information Bottleneck for Gaussian Variablescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2025Distributional Vision-Language Alignment by Cauchy-Schwarz Divergencecited by 7
  2. 2025DocVXQA: Context-Aware Visual Explanations for Document Question Answeringcited by 5
  3. 2024Structured IB: Improving Information Bottleneck with Structured Feature Learningcited by 3
  4. 2025A Recipe for Causal Graph Regression: Confounding Effects Revisitedcited by 3
  5. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  6. 2025Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communicationcited by 1
  7. 2024Generalized Cauchy-Schwarz Divergence and Its Deep Learning Applicationscited by 0
  8. 2025Cross-Modal Retrieval with Cauchy-Schwarz Divergencecited by 0
  9. 2026GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compressioncited by 0