# Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers

arXiv: https://arxiv.org/abs/2405.20145 · HTML: https://mlsanity.com/paper/2405.20145/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 14 of 14)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Dependency Parsing as Head Selection](https://mlsanity.com/paper/1606.01280/) (2016)
- [Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing](https://mlsanity.com/paper/1908.07448/) (2019)
- [nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources](https://mlsanity.com/paper/2309.02373/) (2023)
- [BERT in Plutarch's Shadows](https://mlsanity.com/paper/2211.05673/) (2022)
- [Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation](https://mlsanity.com/paper/2308.13116/) (2023)
- [A Study on Transformer Configuration and Training Objective](https://mlsanity.com/paper/2205.10505/) (2022)
- [From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding](https://mlsanity.com/paper/2305.14571/) (2023)

## Built on by (most cited first, 1 of 1)

- [Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages](https://mlsanity.com/paper/2506.17715/) (2025)

## Abstract (arXiv)

Historical languages present unique challenges to the NLP community, with one prominent hurdle being the limited resources available in their closed corpora. This work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages. For PoS and morphological tagging we adapt a hierarchical tokenization method from Sun et al. (2023) and combine it with the advantages of the DeBERTa-V3 architecture, enabling our models to efficiently learn from every character in the training data. We also demonstrate the effectiveness of character-level T5 models on the lemmatization task. Pre-trained from scratch with limited data, our models achieved first place in the constrained subtask, nearly reaching the performance levels of the unconstrained task's winner. Our code is available at https://github.com/bowphs/SIGTYP-2024-hierarchical-transformers