# A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950

arXiv: https://arxiv.org/abs/2503.19844 · HTML: https://mlsanity.com/paper/2503.19844/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 41 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 10 of 10)

- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Is Word Segmentation Necessary for Deep Learning of Chinese Representations?](https://mlsanity.com/paper/1905.05526/) (2019)
- [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://mlsanity.com/paper/1906.11455/) (2019)
- [Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs](https://mlsanity.com/paper/2412.15993/) (2024)
- [Yes but.. Can ChatGPT Identify Entities in Historical Documents?](https://mlsanity.com/paper/2303.17322/) (2023)
- [On Classification with Large Language Models in Cultural Analytics](https://mlsanity.com/paper/2410.12029/) (2024)
- [A Methodology for Studying Linguistic and Cultural Change in China, 1900-1950](https://mlsanity.com/paper/2502.04286/) (2025)

## Built on by (most cited first, 1 of 1)

- [Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study](https://mlsanity.com/paper/2606.22009/) (2026)

## Abstract (arXiv)

This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER) on Chinese texts from 1900 to 1950. Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes. Using a sample dataset from the Shanghai Library Republican Journal corpus, traditional tools such as Jieba and spaCy are compared to LLMs, including GPT-4o, Claude 3.5, and the GLM series. The results show that LLMs outperform traditional methods in all metrics, albeit at considerably higher computational costs, highlighting a trade-off between accuracy and efficiency. Additionally, LLMs better handle genre-specific challenges such as poetry and temporal variations (i.e., pre-1920 versus post-1920 texts), demonstrating that their contextual learning capabilities can advance NLP approaches to historical texts by reducing the need for domain-specific training data.