# Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators

arXiv: https://arxiv.org/abs/2302.09526 · HTML: https://mlsanity.com/paper/2302.09526/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 19 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?](https://mlsanity.com/paper/1812.10004/) (2018)
- [Confidence Intervals and Hypothesis Testing for High-Dimensional Regression](https://mlsanity.com/paper/1306.3171/) (2013)
- [The jamming transition as a paradigm to understand the loss landscape of deep neural networks](https://mlsanity.com/paper/1809.09349/) (2018)
- [Implicit Regularization in ReLU Networks with the Square Loss](https://mlsanity.com/paper/2012.05156/) (2020)
- [Semi-supervised Inference: General Theory and Estimation of Means](https://mlsanity.com/paper/1606.07268/) (2016)
- [De-biasing the Lasso: Optimal Sample Size for Gaussian Designs](https://mlsanity.com/paper/1508.02757/) (2015)
- [Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance](https://mlsanity.com/paper/1805.10407/) (2018)
- [Efficient and Adaptive Linear Regression in Semi-Supervised Settings](https://mlsanity.com/paper/1701.04889/) (2017)
- [Generalizing semi-supervised generative adversarial networks to regression using feature contrasting](https://mlsanity.com/paper/1811.11269/) (2018)
- [Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning](https://mlsanity.com/paper/2010.05517/) (2020)
- [De-Biasing The Lasso With Degrees-of-Freedom Adjustment](https://mlsanity.com/paper/1902.08885/) (2019)
- [On the prediction loss of the lasso in the partially labeled setting](https://mlsanity.com/paper/1606.06179/) (2016)
- [Semi-Supervised Empirical Risk Minimization: Using unlabeled data to improve prediction](https://mlsanity.com/paper/2009.00606/) (2020)

## Built on by (most cited first, 1 of 1)

- [Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift](https://mlsanity.com/paper/2306.16406/) (2023)

## Abstract (arXiv)

We present a methodology for using unlabeled data to design semi-supervised learning (SSL) methods that improve the predictive performance of supervised learning for regression tasks. The main idea is to design different mechanisms for integrating the unlabeled data, and include in each of them a mixing parameter α, controlling the weight given to the unlabeled data. Focusing on Generalized Linear Models (GLM) and linear interpolators classes of models, we analyze the characteristics of different mixing mechanisms, and prove that it is consistently beneficial to integrate the unlabeled data with some nonzero mixing ratio α>0, in terms of predictive performance. Moreover, we provide a rigorous framework to estimate the best mixing ratio where mixed-SSL delivers the best predictive performance, while using the labeled and unlabeled data on hand. The effectiveness of our methodology in delivering substantial improvement compared to the standard supervised models, in a variety of settings, is demonstrated empirically through extensive simulation, providing empirical support for our theoretical analysis. We also demonstrate the applicability of our methodology (with some heuristic modifications) to improve more complex models, such as deep neural networks, in real-world regression tasks