mlsanityData to 14 Oct 2026

Is Supervised Learning Really That Different from Unsupervised?

arXiv 2505.11006 ↗First posted 16 May 2025stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that - in contrast to cross-validation - can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate that versions of linear and kernel ridge regression, smoothing splines, k-nearest neighbors, random forests, and neural networks, trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 102 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

57 of 102 references.

References · 57

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2015Variational Inference with Normalizing Flowscited by 2,170
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  12. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
38 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  2. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  3. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  6. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 282
  7. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  8. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  9. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  10. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  11. 2019Deep Self-Learning From Noisy Labelscited by 122
  12. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  13. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  14. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  15. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  16. 2020Computing the Testing Error without a Testing Setcited by 54
  17. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  18. 2017Early Stopping without a Validation Setcited by 39
  19. 2021Neural Architecture Search with Random Labelscited by 33
  20. 2021KNAS: Green Neural Architecture Searchcited by 33
  21. 2023A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learningcited by 33
  22. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  23. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 31
  24. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  25. 2020Speedy Performance Estimation for Neural Architecture Searchcited by 24
  26. 2015Early Stopping is Nonparametric Variational Inferencecited by 23
  27. 2025Early Stopping Against Label Noise Without Validation Datacited by 21
  28. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  29. 2020A Bayesian Perspective on Training Speed and Model Selectioncited by 19
  30. 2024Energy-based Automated Model Evaluationcited by 17
  31. 2024Vision-Language Models are Strong Noisy Label Detectorscited by 16
  32. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  33. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  34. 2017From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimationcited by 12
  35. 2018Leveraging Random Label Memorization for Unsupervised Pre-Trainingcited by 12
  36. 2022Predicting the generalization gap in neural networks using topological data analysiscited by 9
  37. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  38. 2024Tackling Noisy Labels with Network Parameter Additive Decompositioncited by 8

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.