mlsanityData to 7 Oct 2026

TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages

arXiv 2404.12845 ↗First posted 19 Apr 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present our submission to the unconstrained subtask of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages for morphological annotation, POS-tagging, lemmatization, character- and word-level gap-filling. We developed a simple, uniform, and computationally lightweight approach based on the adapters framework using parameter-efficient fine-tuning. We applied the same adapter-based approach uniformly to all tasks and 16 languages by fine-tuning stacked language- and task-specific adapters. Our submission obtained an overall second place out of three submissions, with the first place in word-level gap-filling. Our results show the feasibility of adapting language models pre-trained on modern languages to historical and ancient languages via adapter training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

15 of 24 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  11. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  12. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
3 more
  1. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  2. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  3. 2020Latin BERT: A Contextual Language Model for Classical Philologycited by 43

Built on by

Not cited yet.