# Yes but.. Can ChatGPT Identify Entities in Historical Documents?

arXiv: https://arxiv.org/abs/2303.17322 · HTML: https://mlsanity.com/paper/2303.17322/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [hmBERT: Historical Multilingual Language Models for Named Entity Recognition](https://mlsanity.com/paper/2205.15575/) (2022)
- [Optical Character Recognition of 19th Century Classical Commentaries: the Current State of Affairs](https://mlsanity.com/paper/2110.06817/) (2021)
- [Page Layout Analysis of Text-heavy Historical Documents: a Comparison of Textual and Visual Approaches](https://mlsanity.com/paper/2212.13924/) (2022)
- [Towards Robust Named Entity Recognition for Historic German](https://mlsanity.com/paper/1906.07592/) (2019)

## Built on by (most cited first, 10 of 10)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents](https://mlsanity.com/paper/2504.00414/) (2025)
- [Artificial Intelligence in archival and historical scholarship workflow: HTS and ChatGPT](https://mlsanity.com/paper/2308.02044/) (2023)
- [Validating ChatGPT Facts through RDF Knowledge Graphs and Sentence Similarity](https://mlsanity.com/paper/2311.04524/) (2023)
- [A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950](https://mlsanity.com/paper/2503.19844/) (2025)
- [Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM](https://mlsanity.com/paper/2506.02589/) (2025)
- [High Accuracy Location Information Extraction from Social Network Texts Using Natural Language Processing](https://mlsanity.com/paper/2308.16615/) (2023)
- [Token and Span Classification for Entity Recognition in French Historical Encyclopedias](https://mlsanity.com/paper/2506.02872/) (2025)
- [CodeNER: Code Prompting for Named Entity Recognition](https://mlsanity.com/paper/2507.20423/) (2025)
- [Named Entity Recognition of Historical Texts via Large Language Model](https://mlsanity.com/paper/2508.18090/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have been leveraged for several years now, obtaining state-of-the-art performance in recognizing entities from modern documents. For the last few months, the conversational agent ChatGPT has "prompted" a lot of interest in the scientific community and public due to its capacity of generating plausible-sounding answers. In this paper, we explore this ability by probing it in the named entity recognition and classification (NERC) task in primary sources (e.g., historical newspapers and classical commentaries) in a zero-shot manner and by comparing it with state-of-the-art LM-based systems. Our findings indicate several shortcomings in identifying entities in historical text that range from the consistency of entity annotation guidelines, entity complexity, and code-switching, to the specificity of prompting. Moreover, as expected, the inaccessibility of historical archives to the public (and thus on the Internet) also impacts its performance.