mlsanityData to 14 Oct 2026

Self-supervised Regularization for Text Classification

arXiv 2103.05231 ↗First posted 9 Mar 2021cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Text classification is a widely studied problem and has broad applications. In many real-world problems, the number of texts for training classification models is limited, which renders these models prone to overfitting. To address this problem, we propose SSL-Reg, a data-dependent regularization approach based on self-supervised learning (SSL). SSL is an unsupervised learning approach which defines auxiliary tasks on input data without using any human-provided labels and learns data representations by solving these auxiliary tasks. In SSL-Reg, a supervised classification task and an unsupervised SSL task are performed simultaneously. The SSL task is unsupervised, which is defined purely on input texts without using any human-provided labels. Training a model using an SSL task can prevent the model from being overfitted to a limited number of class labels in the classification task. Experiments on 17 text classification datasets demonstrate the effectiveness of our proposed method.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

46 of 62 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2016Layer Normalizationcited by 5,669
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
34 more
  1. 2020Supervised Contrastive Learningcited by 2,829
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  13. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  15. 2015Image-based Recommendations on Styles and Substitutescited by 706
  16. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  17. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
  18. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  19. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  20. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  21. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  22. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  23. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  24. 2020Deep Learning Based Text Classification: A Comprehensive Reviewcited by 141
  25. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  26. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  27. 2020Contrastive Self-supervised Learning for Graph Classificationcited by 68
  28. 2015A C-LSTM Neural Network for Text Classificationcited by 54
  29. 2017Improvements to context based self-supervised learningcited by 32
  30. 2020Contrastive Self-Supervised Learning for Commonsense Reasoningcited by 25
  31. 2020Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigmscited by 23
  32. 2020Pathological Visual Question Answeringcited by 18
  33. 2016Neural Networks for Joint Sentence Classification in Medical Paper Abstractscited by 10
  34. 2019Importance-Aware Learning for Neural Headline Editingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 1; 2025: 0; 2026: 1.

5 papers.

  1. 2022Disconnected Emerging Knowledge Graph Oriented Inductive Link Predictioncited by 7
  2. 2026CGM-JEPA: Learning Consistent Continuous Glucose Monitor Representations via Predictive Self-Supervised Pretrainingcited by 6
  3. 2024IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classificationcited by 4
  4. 2022Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classificationcited by 3
  5. 2023APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learningcited by 0