mlsanityData to 7 Oct 2026

An Assessment of the Impact of OCR Noise on Language Models

arXiv 2202.00470 ↗First posted 26 Jan 2022cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Neural language models are the backbone of modern-day natural language processing applications. Their use on textual heritage collections which have undergone Optical Character Recognition (OCR) is therefore also increasing. Nevertheless, our understanding of the impact OCR noise could have on language models is still limited. We perform an assessment of the impact OCR noise has on a variety of language models, using data in Dutch, English, French and German. We find that OCR noise poses a significant obstacle to language modelling, with language models increasingly diverging from their noiseless targets as OCR quality lowers. In the presence of small corpora, simpler models including PPMI and Word2Vec consistently outperform transformer-based models in this respect.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 29 of its 55 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

29 of 55 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
17 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  3. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
  4. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  5. 2015Character-Aware Neural Language Modelscited by 129
  6. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  7. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  8. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  9. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  10. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  11. 2015Computational Sociolinguistics: A Surveycited by 35
  12. 2020Which *BERT? A Survey Organizing Contextualized Encoderscited by 26
  13. 2021Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corporacited by 21
  14. 2020Living Machines: A study of atypical animacycited by 13
  15. 2019On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languagescited by 11
  16. 2016Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resourcecited by 7
  17. 2019From the Paft to the Fiiture: a Fully Automatic NMT and Word Embeddings Method for OCR Post-Correctioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024Confidence-Aware Document OCR Error Detectioncited by 2
  2. 2024Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documentscited by 0
  3. 2025The InviTE Corpus: Annotating Invectives in Tudor English Texts for Computational Modelingcited by 0
  4. 2026When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscriptscited by 0
  5. 2026Which papyrus HTR is good enough? Character-error-rate tolerance of four papyrological tasks on Greek textscited by 0