mlsanityData to 7 Oct 2026

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models

arXiv 2501.13397 ↗First posted 23 Jan 2025cs.CL cs.LGcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Masked Language Models (MLMs) have achieved remarkable success in many self-supervised representation learning tasks. MLMs are trained by randomly masking portions of the input sequences with [MASK] tokens and learning to reconstruct the original content based on the remaining context. This paper explores the impact of [MASK] tokens on MLMs. Analytical studies show that masking tokens can introduce the corrupted semantics problem, wherein the corrupted context may convey multiple, ambiguous meanings. This problem is also a key factor affecting the performance of MLMs on downstream tasks. Based on these findings, we propose a novel enhanced-context MLM, ExLM. Our approach expands [MASK] tokens in the input context and models the dependencies between these expanded states. This enhancement increases context capacity and enables the model to capture richer semantic information, effectively mitigating the corrupted semantics problem during pre-training. Experimental results demonstrate that ExLM achieves significant performance improvements in both text modeling and SMILES modeling tasks. Further analysis confirms that ExLM enriches semantic representations through context enhancement, and effectively reduces the semantic multimodality commonly observed in MLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 57 of its 90 references that are in the map. First posted on arXiv 23 Jan 2025.

Builds on

57 of 90 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
38 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  10. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  11. 2017Non-Autoregressive Neural Machine Translationcited by 457
  12. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 373
  13. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  14. 2023Improving Text Embeddings with Large Language Modelscited by 295
  15. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  16. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  17. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  18. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  19. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  20. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  21. 2023Representation Learning with Large Language Models for Recommendationcited by 155
  22. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  23. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  24. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  25. 2018N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Moleculescited by 93
  26. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  27. 2020Spelling Error Correction with Soft-Masked BERTcited by 90
  28. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  29. 2024Repetition Improves Language Model Embeddingscited by 70
  30. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  31. 2017"Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Modelscited by 54
  32. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  33. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  34. 2022Contextual Representation Learning beyond Masked Language Modelingcited by 16
  35. 2022Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 13
  36. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
  37. 2023Representation Deficiency in Masked Language Modelingcited by 10
  38. 2022Pretraining Text Encoders with Adversarial Mixture of Training Signal Generatorscited by 9

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Multi-Modal Soccer Scene Analysis with Masked Pre-Trainingcited by 1
  2. 2026AGE: Adaptive-masking for Graph Embedding in Graph Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
  2. 2021Data Efficient Masked Language Modeling for Vision and Language
  3. 2022Token Dropping for Efficient BERT Pretraining
  4. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
  5. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
  6. 2026MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
  7. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
  8. 2026SAMTok: Representing Any Mask with Two Words