Unlabeled Data Help: Minimax Analysis and Adversarial Robustness

arXiv 2202.06996 ↗First posted 14 Feb 2022stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The recent proposed self-supervised learning (SSL) approaches successfully demonstrate the great potential of supplementing learning algorithms with additional unlabeled data. However, it is still unclear whether the existing SSL algorithms can fully utilize the information of both labelled and unlabeled data. This paper gives an affirmative answer for the reconstruction-based SSL algorithm citeplee2020predicting under several statistical models. While existing literature only focuses on establishing the upper bound of the convergence rate, we provide a rigorous minimax analysis, and successfully justify the rate-optimality of the reconstruction-based SSL algorithm under different data generation models. Furthermore, we incorporate the reconstruction-based SSL into the existing adversarial training algorithms and show that learning from unlabeled data helps improve the robustness.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

51 of 67 references.

References · 51

2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
2020Supervised Contrastive Learningcited by 2,803
2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
2016Colorful Image Colorizationcited by 1,113
2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
2016Practical Black-Box Attacks against Machine Learningcited by 893
2020What Makes for Good Views for Contrastive Learning?cited by 843
2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636

38 more

2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
2019Unlabeled Data Improves Adversarial Robustnesscited by 415
2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
2020Debiased Contrastive Learningcited by 354
2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
2018Adversarially Robust Generalization Requires More Datacited by 318
2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
2020What Should Not Be Contrastive in Contrastive Learningcited by 224
2019Are Labels Required for Improving Adversarial Robustness?cited by 162
2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
2020Adversarial Self-Supervised Contrastive Learningcited by 138
2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
2019Adversarial Training Can Hurt Generalizationcited by 131
2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
2019Deep Neural Network Approximation Theorycited by 92
2020Contrastive Learning with Adversarial Examplescited by 91
2020Precise Tradeoffs in Adversarial Training for Linear Regressioncited by 76
2019Adversarially Robust Generalization Just Requires More Unlabeled Datacited by 75
2017DolphinAtack: Inaudible Voice Commandscited by 65
2018Deep Neural Networks for Estimation and Inferencecited by 62
2015A Century of Portraits: A Visual Historical Record of American High School Yearbookscited by 54
2017Contrastive Learning for Image Captioningcited by 54
2019Robustness to Adversarial Perturbations in Learning from Incomplete Datacited by 49
2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 49
2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
2019Exact minimax risk for linear least squares, and the lower tail of sample covariance matricescited by 34
2020Sharp Statistical Guarantees for Adversarially Robust Gaussian Classificationcited by 27
2020On the Generalization Properties of Adversarial Trainingcited by 23
2020Adversarially Robust Estimate and Risk Analysis in Linear Regressioncited by 22
2017Local nearest neighbour classification with applications to semi-supervised learningcited by 17
2018High-dimensional Linear Discriminant Analysis: Optimality, Adaptive Algorithm, and Missing Datacited by 17
2020Asymptotic Behavior of Adversarial Training in Binary Classificationcited by 14
2021Fundamental Tradeoffs in Distributionally Adversarial Trainingcited by 12
2021Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysiscited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
2022Adversarial Rademacher Complexity of Deep Neural Networkscited by 18
2023Interpolation for Robust Learning: Data Augmentation on Wasserstein Geodesicscited by 3
2024Distributionally and Adversarially Robust Logistic Regression via Intersecting Wasserstein Ballscited by 0