mlsanityData to 7 Oct 2026

Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding

arXiv 2109.06466 ↗First posted 14 Sep 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data. However, it's unclear whether they learn similar representations or they can be effectively combined. In this paper, we show that TAPT and ST can be complementary with simple TFS protocol by following TAPT -> Finetuning -> Self-training (TFS) process. Experimental results show that TFS protocol can effectively utilize unlabeled data to achieve strong combined gains consistently across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. We investigate various semi-supervised settings and consistently show that gains from TAPT and ST can be strongly additive by following TFS procedure. We hope that TFS could serve as an important semi-supervised baseline for future NLP studies.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

25 of 36 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
13 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2020Rethinking Pre-training and Self-trainingcited by 352
  5. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  6. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  7. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  8. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  9. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  10. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  11. 2019Learning to Self-Train for Semi-Supervised Few-Shot Classificationcited by 83
  12. 2020CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackerscited by 43
  13. 2020Uncertainty-aware Self-training for Text Classification with Few Labelscited by 19

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 7; 2024: 3.

12 papers.

  1. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  2. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  3. 2023Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognitioncited by 11
  4. 2022DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generationcited by 5
  5. 2023Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Modelscited by 4
  6. 2023AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classificationcited by 3
  7. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  8. 2022Towards Simple and Efficient Task-Adaptive Pre-training for Text Classificationcited by 1
  9. 2023Ensemble Transfer Learning for Multilingual Coreference Resolutioncited by 0
  10. 2023KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generationcited by 0
  11. 2024Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selectioncited by 0
  12. 2024A Comparative Study of Pre-training and Self-trainingcited by 0