# MASKER: Masked Keyword Regularization for Reliable Text Classification

arXiv: https://arxiv.org/abs/2012.09392 · HTML: https://mlsanity.com/paper/2012.09392/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [Discovering and Mitigating Visual Biases through Keyword Explanation](https://mlsanity.com/paper/2301.11104/) (2023)
- [Object-aware Contrastive Learning for Debiased Scene Representation](https://mlsanity.com/paper/2108.00049/) (2021)
- [Time Is MattEr: Temporal Self-supervision for Video Transformers](https://mlsanity.com/paper/2207.09067/) (2022)
- [Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications](https://mlsanity.com/paper/2305.06522/) (2023)
- [Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble](https://mlsanity.com/paper/2210.11034/) (2022)
- [Bag of Tricks for In-Distribution Calibration of Pretrained Transformers](https://mlsanity.com/paper/2302.06690/) (2023)
- [Breaking the Spurious Causality of Conditional Generation via Fairness Intervention with Corrective Sampling](https://mlsanity.com/paper/2212.02090/) (2022)
- [SOUL: Towards Sentiment and Opinion Understanding of Language](https://mlsanity.com/paper/2310.17924/) (2023)
- [Learning Shortcuts: On the Misleading Promise of NLU in Language Models](https://mlsanity.com/paper/2401.09615/) (2024)
- [RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training](https://mlsanity.com/paper/2312.04032/) (2023)
- [Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2301.11660/) (2023)
- [Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP](https://mlsanity.com/paper/2310.14849/) (2023)
- [BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification](https://mlsanity.com/paper/2311.16083/) (2023)
- [TACIT: A Target-Agnostic Feature Disentanglement Framework for Cross-Domain Text Classification](https://mlsanity.com/paper/2312.17263/) (2023)
- [Out-of-Category Document Identification Using Target-Category Names as Weak Supervision](https://mlsanity.com/paper/2111.12796/) (2021)
- [Task-guided Disentangled Tuning for Pretrained Language Models](https://mlsanity.com/paper/2203.11431/) (2022)
- [MASKDROID: Robust Android Malware Detection with Masked Graph Representations](https://mlsanity.com/paper/2409.19594/) (2024)
- [Relation-based Counterfactual Data Augmentation and Contrastive Learning for Robustifying Natural Language Inference Models](https://mlsanity.com/paper/2410.20710/) (2024)
- [Pseudo-OOD training for robust language models](https://mlsanity.com/paper/2210.09132/) (2022)
- [Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals](https://mlsanity.com/paper/2210.11805/) (2022)
- [Continuously Reliable Detection of New-Normal Misinformation: Semantic Masking and Contrastive Smoothing in High-Density Latent Regions](https://mlsanity.com/paper/2301.07981/) (2023)
- [Improving the Reusability of Pre-trained Language Models in Real-world Applications](https://mlsanity.com/paper/2307.10457/) (2023)
- [The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification](https://mlsanity.com/paper/2505.06624/) (2025)
- [KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training](https://mlsanity.com/paper/2609.00082/) (2026)
- [OFBD: Object-Focused Background Debiasing for Long-Tailed Learning](https://mlsanity.com/paper/2609.37331/) (2026)

## Abstract (arXiv)

Pre-trained language models have achieved state-of-the-art accuracies on various text classification tasks, e.g., sentiment analysis, natural language inference, and semantic textual similarity. However, the reliability of the fine-tuned text classifiers is an often underlooked performance criterion. For instance, one may desire a model that can detect out-of-distribution (OOD) samples (drawn far from training distribution) or be robust against domain shifts. We claim that one central obstacle to the reliability is the over-reliance of the model on a limited number of keywords, instead of looking at the whole context. In particular, we find that (a) OOD samples often contain in-distribution keywords, while (b) cross-domain samples may not always contain keywords; over-relying on the keywords can be problematic for both cases. In light of this observation, we propose a simple yet effective fine-tuning method, coined masked keyword regularization (MASKER), that facilitates context-based prediction. MASKER regularizes the model to reconstruct the keywords from the rest of the words and make low-confidence predictions without enough context. When applied to various pre-trained language models (e.g., BERT, RoBERTa, and ALBERT), we demonstrate that MASKER improves OOD detection and cross-domain generalization without degrading classification accuracy. Code is available at https://github.com/alinlab/MASKER.