# Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation

arXiv: https://arxiv.org/abs/2104.05964 · HTML: https://mlsanity.com/paper/2104.05964/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019)
- [Restoring ancient text using deep learning: a case study on Greek epigraphy](https://mlsanity.com/paper/1910.06262/) (2019)
- [KuroNet: Pre-Modern Japanese Kuzushiji Character Recognition with Deep Learning](https://mlsanity.com/paper/1910.09433/) (2019)
- [Convolutional Neural Networks for Page Segmentation of Historical Document Images](https://mlsanity.com/paper/1704.01474/) (2017)
- [Ancient-Modern Chinese Translation with a Large Training Dataset](https://mlsanity.com/paper/1808.03738/) (2018)
- [An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization](https://mlsanity.com/paper/1806.05210/) (2018)
- [Automatic Translating between Ancient Chinese and Contemporary Chinese with Limited Aligned Corpora](https://mlsanity.com/paper/1803.01557/) (2018)

## Built on by (most cited first, 9 of 9)

- [HistRED: A Historical Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/2307.04285/) (2023)
- [Translating Hanja Historical Documents to Contemporary Korean and English](https://mlsanity.com/paper/2205.10019/) (2022)
- [HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea](https://mlsanity.com/paper/2210.05112/) (2022)
- [DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms](https://mlsanity.com/paper/2503.03644/) (2025)
- [HERITAGE: An End-to-End Web Platform for Processing Korean Historical Documents in Hanja](https://mlsanity.com/paper/2501.11951/) (2025)
- [Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach](https://mlsanity.com/paper/2403.06682/) (2024)
- [Review of Computational Epigraphy](https://mlsanity.com/paper/2406.06570/) (2024)
- [Shared Heritage, Distinct Writing: Rethinking Resource Selection for East Asian Historical Documents](https://mlsanity.com/paper/2411.04822/) (2024)
- [Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2607.21936/) (2026)

## Abstract (arXiv)

Understanding voluminous historical records provides clues on the past in various aspects, such as social and political issues and even natural science facts. However, it is generally difficult to fully utilize the historical records, since most of the documents are not written in a modern language and part of the contents are damaged over time. As a result, restoring the damaged or unrecognizable parts as well as translating the records into modern languages are crucial tasks. In response, we present a multi-task learning approach to restore and translate historical documents based on a self-attention mechanism, specifically utilizing two Korean historical records, ones of the most voluminous historical records in the world. Experimental results show that our approach significantly improves the accuracy of the translation task than baselines without multi-task learning. In addition, we present an in-depth exploratory analysis on our translated results via topic modeling, uncovering several significant historical events.