# Applying the Transformer to Character-level Transduction

arXiv: https://arxiv.org/abs/2005.10213 · HTML: https://mlsanity.com/paper/2005.10213/

## Key facts

- Cited by 43 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 30 of 43)

- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models](https://mlsanity.com/paper/2104.06483/) (2021)
- [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://mlsanity.com/paper/2310.15113/) (2023)
- [On Biasing Transformer Attention Towards Monotonicity](https://mlsanity.com/paper/2104.03945/) (2021)
- [Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers](https://mlsanity.com/paper/2410.21013/) (2024)
- [Morphological Reinflection with Multiple Arguments: An Extended Annotation schema and a Georgian Case Study](https://mlsanity.com/paper/2203.08527/) (2022)
- [Morphological Processing of Low-Resource Languages: Where We Are and What's Next](https://mlsanity.com/paper/2203.08909/) (2022)
- [Text normalization for low-resource languages: the case of Ligurian](https://mlsanity.com/paper/2206.07861/) (2022)
- [Connecting the Persian-speaking World through Transliteration](https://mlsanity.com/paper/2502.20047/) (2025)
- [ParsTranslit: Truly Versatile Tajik-Farsi Transliteration](https://mlsanity.com/paper/2510.07520/) (2025)
- [ParaNames: A Massively Multilingual Entity Name Corpus](https://mlsanity.com/paper/2202.14035/) (2022)
- [Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection](https://mlsanity.com/paper/2305.13658/) (2023)
- [An Investigation of Noise in Morphological Inflection](https://mlsanity.com/paper/2305.16581/) (2023)
- [Searching for Search Errors in Neural Morphological Inflection](https://mlsanity.com/paper/2102.08424/) (2021)
- [Computational Morphology with Neural Network Approaches](https://mlsanity.com/paper/2105.09404/) (2021)
- [A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection](https://mlsanity.com/paper/2210.12321/) (2022)
- [Morphological Inflection: A Reality Check](https://mlsanity.com/paper/2305.15637/) (2023)
- [Multilingual context-based pronunciation learning for Text-to-Speech](https://mlsanity.com/paper/2307.16709/) (2023)
- [Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans](https://mlsanity.com/paper/2602.14100/) (2026)
- [Falling Through the Gaps: Neural Architectures as Models of Morphological Rule Learning](https://mlsanity.com/paper/2105.03710/) (2021)
- [Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction](https://mlsanity.com/paper/2106.12698/) (2021)
- [Transformers on Multilingual Clause-Level Morphology](https://mlsanity.com/paper/2211.01736/) (2022)
- [Comparison of Current Approaches to Lemmatization: A Case Study in Estonian](https://mlsanity.com/paper/2404.15003/) (2024)
- [Can a Neural Model Guide Fieldwork? A Case Study on Morphological Data Collection](https://mlsanity.com/paper/2409.14628/) (2024)
- [chDzDT: Word-level morphology-aware language model for Algerian social media text](https://mlsanity.com/paper/2509.01772/) (2025)
- [When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology](https://mlsanity.com/paper/2605.20558/) (2026)
- [BERT Cannot Align Characters](https://mlsanity.com/paper/2109.09700/) (2021)
- [Learning to pronounce as measuring cross-lingual joint orthography-phonology complexity](https://mlsanity.com/paper/2202.00794/) (2022)
- [Cross-lingual Inflection as a Data Augmentation Method for Parsing](https://mlsanity.com/paper/2205.09350/) (2022)

## Abstract (arXiv)

The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. Yet for character-level transduction tasks, e.g. morphological inflection generation and historical text normalization, there are few works that outperform recurrent models using the transformer. In an empirical study, we uncover that, in contrast to recurrent sequence-to-sequence models, the batch size plays a crucial role in the performance of the transformer on character-level tasks, and we show that with a large enough batch size, the transformer does indeed outperform recurrent models. We also introduce a simple technique to handle feature-guided character-level transduction that further improves performance. With these insights, we achieve state-of-the-art performance on morphological inflection and historical text normalization. We also show that the transformer outperforms a strong baseline on two other character-level transduction tasks: grapheme-to-phoneme conversion and transliteration.