# Character-Level Translation with Self-attention

arXiv: https://arxiv.org/abs/2004.14788 · HTML: https://mlsanity.com/paper/2004.14788/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 12 of 12)

- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023)
- [MambaByte: Token-free Selective State Space Model](https://mlsanity.com/paper/2401.13660/) (2024)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving](https://mlsanity.com/paper/2509.02754/) (2025)
- [The boundaries of meaning: a case study in neural machine translation](https://mlsanity.com/paper/2210.00613/) (2022)
- [Character-level NMT and language similarity](https://mlsanity.com/paper/2308.04398/) (2023)
- [Deterministic Reversible Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2406.02517/) (2024)

## Abstract (arXiv)

We explore the suitability of self-attention models for character-level neural machine translation. We test the standard transformer model, as well as a novel variant in which the encoder block combines information from nearby characters using convolutions. We perform extensive experiments on WMT and UN datasets, testing both bilingual and multilingual translation to English using up to three input languages (French, Spanish, and Chinese). Our transformer variant consistently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level alignments.