mlsanityData to 14 Oct 2026

Understanding The Robustness of Self-supervised Learning Through Topic Modeling

arXiv 2203.03539 ↗First posted 2 Feb 2022cs.CL cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning has significantly improved the performance of many NLP tasks. However, how can self-supervised learning discover useful representations, and why is it better than traditional approaches such as probabilistic models are still largely unknown. In this paper, we focus on the context of topic modeling and highlight a key advantage of self-supervised learning - when applied to data generated by topic models, self-supervised learning can be oblivious to the specific model, and hence is less susceptible to model misspecification. In particular, we prove that commonly used self-supervised objectives based on reconstruction or contrastive samples can both recover useful posterior information for general topic models. Empirically, we show that the same objectives can perform on par with posterior inference using the correct model, while outperforming posterior inference using misspecified models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

31 of 54 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
19 more
  1. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  2. 2019Contrastive Multiview Codingcited by 1,301
  3. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  4. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  5. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  6. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  8. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  9. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  10. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  11. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  12. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  13. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  14. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  15. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 72
  16. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  17. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  18. 2021Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysiscited by 9
  19. 2016Provable Algorithms for Inference in Topic Modelscited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  2. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  3. 2023Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditionscited by 3