# Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

arXiv: https://arxiv.org/abs/2109.04513 · HTML: https://mlsanity.com/paper/2109.04513/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Restoring ancient text using deep learning: a case study on Greek epigraphy](https://mlsanity.com/paper/1910.06262/) (2019)
- [Restoration of Fragmentary Babylonian Texts Using Recurrent Neural Networks](https://mlsanity.com/paper/2003.01912/) (2020)
- [Language and Dialect Identification of Cuneiform Texts](https://mlsanity.com/paper/1903.01891/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank](https://mlsanity.com/paper/2205.04086/) (2022)
- [CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models](https://mlsanity.com/paper/2408.17428/) (2024)
- [Reviving Cultural Heritage: A Novel Approach for Comprehensive Historical Document Restoration](https://mlsanity.com/paper/2507.05108/) (2025)
- [Deep Aramaic: Towards a Synthetic Data Paradigm Enabling Machine Learning in Epigraphy](https://mlsanity.com/paper/2310.07310/) (2023)
- [BIRD: Bronze Inscription Restoration and Dating](https://mlsanity.com/paper/2511.01589/) (2025)
- [Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach](https://mlsanity.com/paper/2403.06682/) (2024)
- [Review of Computational Epigraphy](https://mlsanity.com/paper/2406.06570/) (2024)
- [Text Restoration of Ancient Documents with Language Models](https://mlsanity.com/paper/2608.28170/) (2026)

## Abstract (arXiv)

We present models which complete missing text given transliterations of ancient Mesopotamian documents, originally written on cuneiform clay tablets (2500 BCE - 100 CE). Due to the tablets' deterioration, scholars often rely on contextual cues to manually fill in missing parts in the text in a subjective and time-consuming process. We identify that this challenge can be formulated as a masked language modelling task, used mostly as a pretraining objective for contextualized language models. Following, we develop several architectures focusing on the Akkadian language, the lingua franca of the time. We find that despite data scarcity (1M tokens) we can achieve state of the art performance on missing tokens prediction (89% hit@5) using a greedy decoding scheme and pretraining on data from other languages and different time periods. Finally, we conduct human evaluations showing the applicability of our models in assisting experts to transcribe texts in extinct languages.