# Train No Evil: Selective Masking for Task-Guided Pre-Training

arXiv: https://arxiv.org/abs/2004.09733 · HTML: https://mlsanity.com/paper/2004.09733/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 30 of 36)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [CLEVE: Contrastive Pre-training for Event Extraction](https://mlsanity.com/paper/2105.14485/) (2021)
- [NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework](https://mlsanity.com/paper/2111.04130/) (2021)
- [On the Influence of Masking Policies in Intermediate Pre-training](https://mlsanity.com/paper/2104.08840/) (2021)
- [Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification](https://mlsanity.com/paper/2305.12463/) (2023)
- [EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data](https://mlsanity.com/paper/2312.15696/) (2023)
- [KidLM: Advancing Language Models for Children -- Early Insights and Future Directions](https://mlsanity.com/paper/2410.03884/) (2024)
- [Scheduled Multi-task Learning for Neural Chat Translation](https://mlsanity.com/paper/2205.03766/) (2022)
- [CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models](https://mlsanity.com/paper/2102.03752/) (2021)
- [Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning](https://mlsanity.com/paper/2108.00356/) (2021)
- [Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis](https://mlsanity.com/paper/2201.02026/) (2022)
- [APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning](https://mlsanity.com/paper/2212.09282/) (2022)
- [Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords](https://mlsanity.com/paper/2307.07160/) (2023)
- [How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?](https://mlsanity.com/paper/2401.17514/) (2024)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Continual Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2212.09097/) (2022)
- [Difference-Masking: Choosing What to Mask in Continued Pretraining](https://mlsanity.com/paper/2305.14577/) (2023)
- [Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base](https://mlsanity.com/paper/2204.07994/) (2022)
- [Using Selective Masking as a Bridge between Pre-training and Fine-tuning](https://mlsanity.com/paper/2211.13815/) (2022)
- [BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition](https://mlsanity.com/paper/2308.08625/) (2023)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA](https://mlsanity.com/paper/2510.04682/) (2025)
- [Task-guided Disentangled Tuning for Pretrained Language Models](https://mlsanity.com/paper/2203.11431/) (2022)
- [SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words](https://mlsanity.com/paper/2204.07779/) (2022)
- [Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways](https://mlsanity.com/paper/2310.17591/) (2023)
- [CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search](https://mlsanity.com/paper/2412.01269/) (2024)
- [MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering](https://mlsanity.com/paper/2112.02839/) (2021)
- [KESA: A Knowledge Enhanced Approach For Sentiment Analysis](https://mlsanity.com/paper/2202.12093/) (2022)
- [An Anchor Learning Approach for Citation Field Learning](https://mlsanity.com/paper/2309.03559/) (2023)

## Abstract (arXiv)

Recently, pre-trained language models mostly follow the pre-train-then-fine-tuning paradigm and have achieved great performance on various downstream tasks. However, since the pre-training stage is typically task-agnostic and the fine-tuning stage usually suffers from insufficient supervised data, the models cannot always well capture the domain-specific and task-specific patterns. In this paper, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine-tuning. In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns. Specifically, we design a method to measure the importance of each token in sequences and selectively mask the important tokens. Experimental results on two sentiment analysis tasks show that our method can achieve comparable or even better performance with less than 50% of computation cost, which indicates our method is both effective and efficient. The source code of this paper can be obtained from https://github.com/thunlp/SelectiveMasking.