# Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling

arXiv: https://arxiv.org/abs/2105.11872 · HTML: https://mlsanity.com/paper/2105.11872/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Robust Encodings: A Framework for Combating Adversarial Typos](https://mlsanity.com/paper/2005.01229/) (2020)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation](https://mlsanity.com/paper/1902.01509/) (2019)
- [A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning](https://mlsanity.com/paper/1907.01256/) (2019)
- [How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?](https://mlsanity.com/paper/1704.04441/) (2017)
- [Misspelling Oblivious Word Embeddings](https://mlsanity.com/paper/1905.09755/) (2019)
- [NAT: Noise-Aware Training for Robust Neural Sequence Labeling](https://mlsanity.com/paper/2005.07162/) (2020)
- [Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation](https://mlsanity.com/paper/1906.01569/) (2019)
- [Artificial Error Generation with Machine Translation and Syntactic Patterns](https://mlsanity.com/paper/1707.05236/) (2017)
- [Wronging a Right: Generating Better Errors to Improve Grammatical Error Detection](https://mlsanity.com/paper/1810.00668/) (2018)
- [From the Paft to the Fiiture: a Fully Automatic NMT and Word Embeddings Method for OCR Post-Correction](https://mlsanity.com/paper/1910.05535/) (2019)
- [Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks](https://mlsanity.com/paper/1610.07796/) (2016)
- [Assessing the Tolerance of Neural Machine Translation Systems Against Speech Recognition Errors](https://mlsanity.com/paper/1904.10997/) (2019)

## Built on by (most cited first, 8 of 8)

- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.10169/) (2023)
- [Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation](https://mlsanity.com/paper/2407.18562/) (2024)
- [Noise-Aware Named Entity Recognition for Historical VET Documents](https://mlsanity.com/paper/2601.00488/) (2026)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)
- [Two-Step Occupation Coding](https://mlsanity.com/paper/2607.20101/) (2026)

## Abstract (arXiv)

Despite recent advances, standard sequence labeling systems often fail when processing noisy user-generated text or consuming the output of an Optical Character Recognition (OCR) process. In this paper, we improve the noise-aware training method by proposing an empirical error generation approach that employs a sequence-to-sequence model trained to perform translation from error-free to erroneous text. Using an OCR engine, we generated a large parallel text corpus for training and produced several real-world noisy sequence labeling benchmarks for evaluation. Moreover, to overcome the data sparsity problem that exacerbates in the case of imperfect textual input, we learned noisy language model-based embeddings. Our approach outperformed the baseline noise generation and error correction techniques on the erroneous sequence labeling data sets. To facilitate future research on robustness, we make our code, embeddings, and data conversion scripts publicly available.