Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers
Not on the map. Why?
Abstract (arXiv)
Historical languages present unique challenges to the NLP community, with one prominent hurdle being the limited resources available in their closed corpora. This work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages. For PoS and morphological tagging we adapt a hierarchical tokenization method from Sun et al. (2023) and combine it with the advantages of the DeBERTa-V3 architecture, enabling our models to efficiently learn from every character in the training data. We also demonstrate the effectiveness of character-level T5 models on the lemmatization task. Pre-trained from scratch with limited data, our models achieved first place in the constrained subtask, nearly reaching the performance levels of the unconstrained task's winner. Our code is available at https://github.com/bowphs/SIGTYP-2024-hierarchical-transformers
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 30 May 2024.
Builds on
14 of 24 references.
References · 14
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
- 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
- 2020Latin BERT: A Contextual Language Model for Classical Philologycited by 43
- 2016Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Losscited by 30
- 2016Dependency Parsing as Head Selectioncited by 13
- 2019Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsingcited by 8
- 2023nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resourcescited by 7
- 2022BERT in Plutarch's Shadowscited by 6
- 2023Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillationcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.