# Exploring Large Language Models for Classical Philology

arXiv: https://arxiv.org/abs/2305.13698 · HTML: https://mlsanity.com/paper/2305.13698/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [GREEK-BERT: The Greeks visiting Sesame Street](https://mlsanity.com/paper/2008.12014/) (2020)
- [Dependency Parsing as Head Selection](https://mlsanity.com/paper/1606.01280/) (2016)
- [Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing](https://mlsanity.com/paper/1908.07448/) (2019)
- [BERT in Plutarch's Shadows](https://mlsanity.com/paper/2211.05673/) (2022)
- [POS tagging, lemmatization and dependency parsing of West Frisian](https://mlsanity.com/paper/2107.07974/) (2021)

## Built on by (most cited first, 10 of 10)

- [Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation](https://mlsanity.com/paper/2308.13116/) (2023)
- [Graecia capta ferum victorem cepit. Detecting Latin Allusions to Ancient Greek Literature](https://mlsanity.com/paper/2308.12008/) (2023)
- ['UFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin](https://mlsanity.com/paper/2404.05839/) (2024)
- [From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication](https://mlsanity.com/paper/2606.01136/) (2026)
- [Nostra Domina at EvaLatin 2024: Improving Latin Polarity Detection through Data Augmentation](https://mlsanity.com/paper/2404.07792/) (2024)
- [The Patrologia Graeca Corpus: OCR, Annotation, and Open Release of Noisy Nineteenth-Century Polytonic Greek Editions](https://mlsanity.com/paper/2603.09470/) (2026)
- [RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering](https://mlsanity.com/paper/2604.20738/) (2026)
- [From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages](https://mlsanity.com/paper/2607.24542/) (2026)
- [Transfer Learning for Named Entity Recognition of Classical Latin through LLM Prompting](https://mlsanity.com/paper/2608.04015/) (2026)
- [Apollo Restore: A Foundation LLM for Historical Greek Optimized for Fill-in-the-Middle Restoration of Ancient Greek Texts](https://mlsanity.com/paper/2609.22455/) (2026)

## Abstract (arXiv)

Recent advances in NLP have led to the creation of powerful language models for many languages including Ancient Greek and Latin. While prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages: we explore (i) encoder-only and encoder-decoder architectures using RoBERTa and T5 as strong model types, and create for each of them (ii) a monolingual Ancient Greek and a multilingual instance that includes Latin and English. We evaluate all models on morphological and syntactic tasks, including lemmatization, which demonstrates the added value of T5's decoding abilities. We further define two probing tasks to investigate the knowledge acquired by models pre-trained on Classical texts. Our experiments provide the first benchmarking analysis of existing models of Ancient Greek. Results show that our models provide significant improvements over the SoTA. The systematic analysis of model types can inform future research in designing language models for Classical languages, including the development of novel generative tasks. We make all our models available as community resources, along with a large curated pre-training corpus for Ancient Greek, to support the creation of a larger, comparable model zoo for Classical Philology. Our models and resources are available at https://github.com/Heidelberg-NLP/ancient-language-models.