# Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation

arXiv: https://arxiv.org/abs/2308.13116 · HTML: https://mlsanity.com/paper/2308.13116/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Morphology Matters: A Multilingual Language Modeling Analysis](https://mlsanity.com/paper/2012.06262/) (2020)
- [Multilingual Representation Distillation with Contrastive Learning](https://mlsanity.com/paper/2210.05033/) (2022)
- [Exploring Large Language Models for Classical Philology](https://mlsanity.com/paper/2305.13698/) (2023)
- [BERT in Plutarch's Shadows](https://mlsanity.com/paper/2211.05673/) (2022)
- [Logion: Machine Learning for Greek Philology](https://mlsanity.com/paper/2305.01099/) (2023)

## Built on by (most cited first, 6 of 6)

- [ALGEN: Few-shot Inversion Attacks on Textual Embeddings using Alignment and Generation](https://mlsanity.com/paper/2502.11308/) (2025)
- [Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers](https://mlsanity.com/paper/2405.20145/) (2024)
- [PhiloBERTA: A Transformer-Based Cross-Lingual Analysis of Greek and Latin Lexicons](https://mlsanity.com/paper/2503.05265/) (2025)
- [From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages](https://mlsanity.com/paper/2607.24542/) (2026)
- [Stoicheia: Character-Level Masked Diffusion for Ancient Greek Textual Restoration, Parsing, and Metrical Scansion](https://mlsanity.com/paper/2608.07249/) (2026)
- [When Do Agents Help? Embedding, LLM and Agentic Alignment of Classical Texts and Their Translations](https://mlsanity.com/paper/2609.33691/) (2026)

## Abstract (arXiv)

Contextual language models have been trained on Classical languages, including Ancient Greek and Latin, for tasks such as lemmatization, morphological tagging, part of speech tagging, authorship attribution, and detection of scribal errors. However, high-quality sentence embedding models for these historical languages are significantly more difficult to achieve due to the lack of training data. In this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text. The state-of-the-art sentence embedding approaches for high-resource languages use massive datasets, but our distillation approach allows our Ancient Greek models to inherit the properties of these models while using a relatively small amount of translated sentence data. We build a parallel sentence dataset using a sentence-embedding alignment method to align Ancient Greek documents with English translations, and use this dataset to train our models. We evaluate our models on translation search, semantic similarity, and semantic retrieval tasks and investigate translation bias. We make our training and evaluation datasets freely available at https://github.com/kevinkrahn/ancient-greek-datasets .