# ALBERTI, a Multilingual Domain Specific Language Model for Poetry Analysis

arXiv: https://arxiv.org/abs/2307.01387 · HTML: https://mlsanity.com/paper/2307.01387/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme](https://mlsanity.com/paper/1807.03491/) (2018)
- [PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation](https://mlsanity.com/paper/2205.12206/) (2022)
- [PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry](https://mlsanity.com/paper/2003.07723/) (2020)
- [A Comparison of Feature-Based and Neural Scansion of Poetry](https://mlsanity.com/paper/1711.00938/) (2017)
- [Semantics of European poetry is shaped by conservative forces: The relationship between poetic meter and meaning in accentual-syllabic verse](https://mlsanity.com/paper/2109.07148/) (2021)

## Built on by (most cited first, 3 of 3)

- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence](https://mlsanity.com/paper/2309.14379/) (2023)
- [Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2502.01491/) (2025)

## Abstract (arXiv)

The computational analysis of poetry is limited by the scarcity of tools to automatically analyze and scan poems. In a multilingual settings, the problem is exacerbated as scansion and rhyme systems only exist for individual languages, making comparative studies very challenging and time consuming. In this work, we present textscAlberti, the first multilingual pre-trained large language model for poetry. Through domain-specific pre-training (DSP), we further trained multilingual BERT on a corpus of over 12 million verses from 12 languages. We evaluated its performance on two structural poetry tasks: Spanish stanza type classification, and metrical pattern prediction for Spanish, English and German. In both cases, textscAlberti outperforms multilingual BERT and other transformers-based models of similar sizes, and even achieves state-of-the-art results for German when compared to rule-based systems, demonstrating the feasibility and effectiveness of DSP in the poetry domain.