# GujiBERT and GujiGPT: Construction of Intelligent Information Processing Foundation Language Models for Ancient Texts

arXiv: https://arxiv.org/abs/2307.05354 · HTML: https://mlsanity.com/paper/2307.05354/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)
- [Generating Major Types of Chinese Classical Poetry in a Uniformed Framework](https://mlsanity.com/paper/2003.11528/) (2020)
- [SikuGPT: A Generative Pre-trained Model for Intelligent Information Processing of Ancient Texts from the Perspective of Digital Humanities](https://mlsanity.com/paper/2304.07778/) (2023)
- [Sentence Segmentation for Classical Chinese Based on LSTM with Radical Embedding](https://mlsanity.com/paper/1810.03479/) (2018)
- [Ancient Chinese Word Segmentation and Part-of-Speech Tagging Using Distant Supervision](https://mlsanity.com/paper/2303.01912/) (2023)

## Built on by (most cited first, 9 of 9)

- [WenyanGPT: A Large Language Model for Classical Chinese Tasks](https://mlsanity.com/paper/2504.20609/) (2025)
- [Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese](https://mlsanity.com/paper/2403.00509/) (2024)
- [TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models](https://mlsanity.com/paper/2407.03937/) (2024)
- [RiverEcho: Real-Time Interactive Digital System for Ancient Yellow River Culture](https://mlsanity.com/paper/2506.21865/) (2025)
- [InteChar: A Unified Oracle Bone Character List for Ancient Chinese Language Modeling](https://mlsanity.com/paper/2508.15791/) (2025)
- [Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition](https://mlsanity.com/paper/2312.15304/) (2023)
- [Named Entity Recognition in Context](https://mlsanity.com/paper/2503.20836/) (2025)
- [ICH-Qwen: A Large Language Model Towards Chinese Intangible Cultural Heritage](https://mlsanity.com/paper/2505.23831/) (2025)
- [Text Restoration of Ancient Documents with Language Models](https://mlsanity.com/paper/2608.28170/) (2026)

## Abstract (arXiv)

In the context of the rapid development of large language models, we have meticulously trained and introduced the GujiBERT and GujiGPT language models, which are foundational models specifically designed for intelligent information processing of ancient texts. These models have been trained on an extensive dataset that encompasses both simplified and traditional Chinese characters, allowing them to effectively handle various natural language processing tasks related to ancient books, including but not limited to automatic sentence segmentation, punctuation, word segmentation, part-of-speech tagging, entity recognition, and automatic translation. Notably, these models have exhibited exceptional performance across a range of validation tasks using publicly available datasets. Our research findings highlight the efficacy of employing self-supervised methods to further train the models using classical text corpora, thus enhancing their capability to tackle downstream tasks. Moreover, it is worth emphasizing that the choice of font, the scale of the corpus, and the initial model selection all exert significant influence over the ultimate experimental outcomes. To cater to the diverse text processing preferences of researchers in digital humanities and linguistics, we have developed three distinct categories comprising a total of nine model variations. We believe that by sharing these foundational language models specialized in the domain of ancient texts, we can facilitate the intelligent processing and scholarly exploration of ancient literary works and, consequently, contribute to the global dissemination of China's rich and esteemed traditional culture in this new era.