# GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text

arXiv: https://arxiv.org/abs/2403.06399 · HTML: https://mlsanity.com/paper/2403.06399/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [Using Interlinear Glosses as Pivot in Low-Resource Multilingual Machine Translation](https://mlsanity.com/paper/1911.02709/) (2019)
- [User-Centric Evaluation of OCR Systems for Kwak'wala](https://mlsanity.com/paper/2302.13410/) (2023)
- [Robust Generalization Strategies for Morpheme Glossing in an Endangered Language Documentation Context](https://mlsanity.com/paper/2311.02777/) (2023)
- [Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning](https://mlsanity.com/paper/2206.05154/) (2022)
- [SIGMORPHON 2023 Shared Task of Interlinear Glossing: Baseline Model](https://mlsanity.com/paper/2303.14234/) (2023)

## Built on by (most cited first, 12 of 12)

- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing](https://mlsanity.com/paper/2406.11085/) (2024)
- [Read it in Two Steps: Translating Extremely Low-Resource Languages with Code-Augmented Grammar Books](https://mlsanity.com/paper/2506.01796/) (2025)
- [Interdisciplinary Research in Conversation: A Case Study in Computational Morphology for Language Documentation](https://mlsanity.com/paper/2509.10644/) (2025)
- [Scaling Participation in Modular AI Systems](https://mlsanity.com/paper/2606.07812/) (2026)
- [CMULAB: An Open-Source Framework for Training and Deployment of Natural Language Processing Models](https://mlsanity.com/paper/2404.02408/) (2024)
- [Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages](https://mlsanity.com/paper/2502.09778/) (2025)
- [Massively Multilingual Joint Segmentation and Glossing](https://mlsanity.com/paper/2601.10925/) (2026)
- [Testing the Limits of Machine Translation from One Book](https://mlsanity.com/paper/2508.06665/) (2025)
- [Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvan](https://mlsanity.com/paper/2603.00923/) (2026)
- [CWoMP: Morpheme Representation Learning for Interlinear Glossing](https://mlsanity.com/paper/2603.18184/) (2026)
- [GlossAssist -- A Tool to Simplify Corpus Creation and Study the Effect of NLP Models in Low-Resource Documentation Settings](https://mlsanity.com/paper/2606.04367/) (2026)

## Abstract (arXiv)

Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages. Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6%. Our pretrained model and dataset are available on Hugging Face.