# Robust Named Entity Recognition with Truecasing Pretraining

arXiv: https://arxiv.org/abs/1912.07095 · HTML: https://mlsanity.com/paper/1912.07095/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2019.

## Built on by (most cited first, 11 of 11)

- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition](https://mlsanity.com/paper/2107.11610/) (2021)
- [Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition](https://mlsanity.com/paper/2105.14980/) (2021)
- [Attack Named Entity Recognition by Entity Boundary Interference](https://mlsanity.com/paper/2305.05253/) (2023)
- [AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs](https://mlsanity.com/paper/2409.09704/) (2024)
- [Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network](https://mlsanity.com/paper/2108.11943/) (2021)
- [Noise-Aware Named Entity Recognition for Historical VET Documents](https://mlsanity.com/paper/2601.00488/) (2026)
- [Empirical Study of Named Entity Recognition Performance Using Distribution-aware Word Embedding](https://mlsanity.com/paper/2109.01636/) (2021)
- [Research on Named Entity Recognition in Improved transformer with R-Drop structure](https://mlsanity.com/paper/2306.08315/) (2023)
- [Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach](https://mlsanity.com/paper/2311.03401/) (2023)
- [How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?](https://mlsanity.com/paper/2404.05088/) (2024)

## Abstract (arXiv)

Although modern named entity recognition (NER) systems show impressive performance on standard datasets, they perform poorly when presented with noisy data. In particular, capitalization is a strong signal for entities in many languages, and even state of the art models overfit to this feature, with drastically lower performance on uncapitalized text. In this work, we address the problem of robustness of NER systems in data with noisy or uncertain casing, using a pretraining objective that predicts casing in text, or a truecaser, leveraging unlabeled data. The pretrained truecaser is combined with a standard BiLSTM-CRF model for NER by appending output distributions to character embeddings. In experiments over several datasets of varying domain and casing quality, we show that our new model improves performance in uncased text, even adding value to uncased BERT embeddings. Our method achieves a new state of the art on the WNUT17 shared task dataset.