# MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

arXiv: https://arxiv.org/abs/2108.13655 · HTML: https://mlsanity.com/paper/2108.13655/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://mlsanity.com/paper/2010.11683/) (2020)
- [Learning to Paraphrase for Question Answering](https://mlsanity.com/paper/1708.06022/) (2017)
- [Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering](https://mlsanity.com/paper/2004.11892/) (2020)
- [XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering](https://mlsanity.com/paper/1905.11471/) (2019)
- [Conditional Augmentation for Aspect Term Extraction via Masked Sequence-to-Sequence Generation](https://mlsanity.com/paper/2004.14769/) (2020)
- [Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond](https://mlsanity.com/paper/2010.12405/) (2020)
- [Soft Gazetteers for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2005.01866/) (2020)
- [UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP](https://mlsanity.com/paper/2004.13240/) (2020)

## Built on by (most cited first, 30 of 42)

- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [Prompt Perturbation Consistency Learning for Robust Language Models](https://mlsanity.com/paper/2402.15833/) (2024)
- [Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models](https://mlsanity.com/paper/2311.00287/) (2023)
- [Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning](https://mlsanity.com/paper/2311.09821/) (2023)
- [PromptNER: Prompt Locating and Typing for Named Entity Recognition](https://mlsanity.com/paper/2305.17104/) (2023)
- [Analysing the Robustness of Vision-Language-Models to Common Corruptions](https://mlsanity.com/paper/2504.13690/) (2025)
- [Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples](https://mlsanity.com/paper/2111.10962/) (2021)
- [Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings](https://mlsanity.com/paper/2201.05575/) (2022)
- [ConNER: Consistency Training for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2211.09394/) (2022)
- [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://mlsanity.com/paper/2211.10330/) (2022)
- [LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2402.14568/) (2024)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER](https://mlsanity.com/paper/2306.00928/) (2023)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP](https://mlsanity.com/paper/2404.00415/) (2024)
- [Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning](https://mlsanity.com/paper/2305.13628/) (2023)
- [ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions](https://mlsanity.com/paper/2406.04286/) (2024)
- [PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks](https://mlsanity.com/paper/2210.08855/) (2022)
- [Entity-to-Text based Data Augmentation for various Named Entity Recognition Tasks](https://mlsanity.com/paper/2210.10343/) (2022)
- [AUC Maximization for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2212.04800/) (2022)
- [BioAug: Conditional Generation based Data Augmentation for Low-Resource Biomedical NER](https://mlsanity.com/paper/2305.10647/) (2023)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations](https://mlsanity.com/paper/2211.08794/) (2022)
- [Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling](https://mlsanity.com/paper/2302.08659/) (2023)
- [VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2402.18374/) (2024)
- [DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models](https://mlsanity.com/paper/2404.05083/) (2024)
- [Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?](https://mlsanity.com/paper/2407.02062/) (2024)
- [LLMs as Data Annotators: How Close Are We to Human Performance](https://mlsanity.com/paper/2504.15022/) (2025)
- [Low-Resource Named Entity Recognition: Can One-vs-All AUC Maximization Help?](https://mlsanity.com/paper/2311.04918/) (2023)

## Abstract (arXiv)

Data augmentation is an effective solution to data scarcity in low-resource scenarios. However, when applied to token-level tasks such as NER, data augmentation methods often suffer from token-label misalignment, which leads to unsatsifactory performance. In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation framework for low-resource NER. To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly conditioning on their labels. Thereby, MELM generates high-quality augmented data with novel entities, which provides rich entity regularity knowledge and boosts NER performance. When training data from multiple languages are available, we also integrate MELM with code-mixing for further improvement. We demonstrate the effectiveness of MELM on monolingual, cross-lingual and multilingual NER across various low-resource levels. Experimental results show that our MELM presents substantial improvement over the baseline methods.