# Transformers on Multilingual Clause-Level Morphology

arXiv: https://arxiv.org/abs/2211.01736 · HTML: https://mlsanity.com/paper/2211.01736/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [Pushing the Limits of Low-Resource Morphological Inflection](https://mlsanity.com/paper/1908.05838/) (2019)
- [Exact Hard Monotonic Attention for Character-Level Transduction](https://mlsanity.com/paper/1905.06319/) (2019)
- [Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects](https://mlsanity.com/paper/2110.06852/) (2021)
- [A Simple Joint Model for Improved Contextual Neural Lemmatization](https://mlsanity.com/paper/1904.02306/) (2019)
- [Two Birds with One Stone: Investigating Invertible Neural Networks for Inverse Problems in Morphology](https://mlsanity.com/paper/1912.05274/) (2019)
- [Character-Level Models versus Morphology in Semantic Role Labeling](https://mlsanity.com/paper/1805.11937/) (2018)

## Built on by (most cited first, 1 of 1)

- [GECTurk: Grammatical Error Correction and Detection Dataset for Turkish](https://mlsanity.com/paper/2309.11346/) (2023)

## Abstract (arXiv)

This paper describes our winning systems in MRL: The 1st Shared Task on Multilingual Clause-level Morphology (EMNLP 2022 Workshop) designed by KUIS AI NLP team. We present our work for all three parts of the shared task: inflection, reinflection, and analysis. We mainly explore transformers with two approaches: (i) training models from scratch in combination with data augmentation, and (ii) transfer learning with prefix-tuning at multilingual morphological tasks. Data augmentation significantly improves performance for most languages in the inflection and reinflection tasks. On the other hand, Prefix-tuning on a pre-trained mGPT model helps us to adapt analysis tasks in low-data and multilingual settings. While transformer architectures with data augmentation achieved the most promising results for inflection and reinflection tasks, prefix-tuning on mGPT received the highest results for the analysis task. Our systems received 1st place in all three tasks in MRL 2022.