mlsanityData to 14 Oct 2026

Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems

arXiv 2211.12685 ↗First posted 23 Nov 2022stat.ML cs.IT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

As one of the central tasks in machine learning, regression finds lots of applications in different fields. An existing common practice for solving regression problems is the mean square error (MSE) minimization approach or its regularized variants which require prior knowledge about the models. Recently, Yi et al., proposed a mutual information based supervised learning framework where they introduced a label entropy regularization which does not require any prior knowledge. When applied to classification tasks and solved via a stochastic gradient descent (SGD) optimization algorithm, their approach achieved significant improvement over the commonly used cross entropy loss and its variants. However, they did not provide a theoretical convergence analysis of the SGD algorithm for the proposed formulation. Besides, applying the framework to regression tasks is nontrivial due to the potentially infinite support set of the label. In this paper, we investigate the regression under the mutual information based supervised learning framework. We first argue that the MSE minimization approach is equivalent to a conditional entropy learning problem, and then propose a mutual information learning formulation for solving regression problems by using a reparameterization technique. For the proposed formulation, we give the convergence analysis of the SGD algorithm for solving it in practice. Finally, we consider a multi-output regression data model where we derive the generalization performance lower bound in terms of the mutual information associated with the underlying data distribution. The result shows that the high dimensionality can be a bless instead of a curse, which is controlled by a threshold. We hope our work will serve as a good starting point for further research on the mutual information based regression.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

34 of 54 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  8. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  9. 2015An Introduction to Matrix Concentration Inequalitiescited by 397
  10. 2017Compressed Sensing using Generative Modelscited by 339
  11. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  12. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
22 more
  1. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  2. 2022Balanced MSE for Imbalanced Visual Regressioncited by 89
  3. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  4. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  5. 2021InfinityGAN: Towards Infinite-Pixel Image Synthesiscited by 61
  6. 2018Blessing of dimensionality: mathematical foundations of the statistical physics of datacited by 55
  7. 2015Robust Regression via Hard Thresholdingcited by 39
  8. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 38
  9. 2022Progressive End-to-End Object Detection in Crowded Scenescited by 31
  10. 2022Unsupervised Image-to-Image Translation with Generative Priorcited by 16
  11. 2018Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimizationcited by 14
  12. 2020On the Information Plane of Autoencoderscited by 10
  13. 2020Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutionscited by 7
  14. 2020High--Dimensional Brain in a High-Dimensional World: Blessing of Dimensionalitycited by 6
  15. 2021Depth from Camera Motion and Object Detectioncited by 6
  16. 2019Trust but Verify: An Information-Theoretic Explanation for the Adversarial Fragility of Machine Learning Systems, and a General Defense against Adversarial Attackscited by 4
  17. 2020Robust Machine Learning via Privacy/Rate-Distortion Theorycited by 4
  18. 2020Derivation of Information-Theoretically Optimal Adversarial Attacks with Applications to Robust Machine Learningcited by 4
  19. 2018Optimal Rates of Sketched-regularized Algorithms for Least-Squares Regression over Hilbert Spacescited by 3
  20. 2021One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networkscited by 3
  21. 2022Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systemscited by 2
  22. 2018Outlier Detection using Generative Models with Theoretical Performance Guaranteescited by 1

Built on by

Not cited yet.