mlsanityData to 7 Oct 2026

Blessing of Class Diversity in Pre-training

arXiv 2209.03447 ↗First posted 7 Sep 2022cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper presents a new statistical analysis aiming to explain the recent superior achievements of the pre-training techniques in natural language processing (NLP). We prove that when the classes of the pre-training task (e.g., different words in the masked language model task) are sufficiently diverse, in the sense that the least singular value of the last linear layer in pre-training (denoted as tildenu) is large, then pre-training can significantly improve the sample efficiency of downstream tasks. Specially, we show the transfer learning excess risk enjoys an Oleft(frac1tildenu √nright) rate, in contrast to the Oleft(frac1√mright) rate in the standard supervised learning. Here, n is the number of pre-training data and m is the number of data in the downstream task, and typically n gg m. Our proof relies on a vector-form Rademacher complexity chain rule for disassembling composite function classes and a modified self-concordance condition. These techniques can be of independent interest.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

26 of 45 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2021Convex optimizationcited by 2,479
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  12. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
14 more
  1. 2015The Benefit of Multitask Representation Learningcited by 194
  2. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  3. 2020Provable Meta-Learning of Linear Representationscited by 161
  4. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  5. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  6. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  7. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  8. 2018Logistic Regression: The Importance of Being Impropercited by 40
  9. 2013Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regressioncited by 32
  10. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  11. 2020For self-supervised learning, Rationality implies generalization, provablycited by 22
  12. 2017High-dimensional classification by sparse logistic regressioncited by 19
  13. 2020Strength from Weakness: Fast Learning Using Weak Supervisioncited by 18
  14. 2022Active Multi-Task Representation Learningcited by 13

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  2. 2023On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-trainingcited by 14
  3. 2024Task Diversity Shortens the ICL Plateaucited by 5
  4. 2024Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspectivecited by 0