# Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

arXiv: https://arxiv.org/abs/2409.16934 · HTML: https://mlsanity.com/paper/2409.16934/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Analyzing Encoded Concepts in Transformer Language Models](https://mlsanity.com/paper/2206.13289/) (2022)
- [On the Linguistic Representational Power of Neural Machine Translation Models](https://mlsanity.com/paper/1911.00317/) (2019)
- [Discovery of Natural Language Concepts in Individual Units of CNNs](https://mlsanity.com/paper/1902.07249/) (2019)
- [Label Supervised LLaMA Finetuning](https://mlsanity.com/paper/2310.01208/) (2023)
- [hmBERT: Historical Multilingual Language Models for Named Entity Recognition](https://mlsanity.com/paper/2205.15575/) (2022)
- [An Assessment of the Impact of OCR Noise on Language Models](https://mlsanity.com/paper/2202.00470/) (2022)
- [QURATOR: Innovative Technologies for Content and Data Curation](https://mlsanity.com/paper/2004.12195/) (2020)

## Abstract (arXiv)

This paper investigates the presence of OCR-sensitive neurons within the Transformer architecture and their influence on named entity recognition (NER) performance on historical documents. By analysing neuron activation patterns in response to clean and noisy text inputs, we identify and then neutralise OCR-sensitive neurons to improve model performance. Based on two open access large language models (Llama2 and Mistral), experiments demonstrate the existence of OCR-sensitive regions and show improvements in NER performance on historical newspapers and classical commentaries, highlighting the potential of targeted neuron modulation to improve models' performance on noisy text.