mlsanityData to 7 Oct 2026

Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

arXiv 2409.16934 ↗First posted 25 Sep 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates the presence of OCR-sensitive neurons within the Transformer architecture and their influence on named entity recognition (NER) performance on historical documents. By analysing neuron activation patterns in response to clean and noisy text inputs, we identify and then neutralise OCR-sensitive neurons to improve model performance. Based on two open access large language models (Llama2 and Mistral), experiments demonstrate the existence of OCR-sensitive regions and show improvements in NER performance on historical newspapers and classical commentaries, highlighting the potential of targeted neuron modulation to improve models' performance on noisy text.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

30 of 53 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Mistral 7Bcited by 3,065
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  12. 2023Language Models Represent Space and Timecited by 306
18 more
  1. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  2. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  3. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  4. 2015Visualizing and Understanding Recurrent Networkscited by 158
  5. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  6. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  7. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  8. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  9. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  10. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  11. 2021Named Entity Recognition and Classification on Historical Documents: A Surveycited by 47
  12. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  13. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  14. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21
  15. 2023Label Supervised LLaMA Finetuningcited by 20
  16. 2022hmBERT: Historical Multilingual Language Models for Named Entity Recognitioncited by 8
  17. 2022An Assessment of the Impact of OCR Noise on Language Modelscited by 5
  18. 2020QURATOR: Innovative Technologies for Content and Data Curationcited by 4

Built on by

Not cited yet.