# Revisiting Character-Based Neural Machine Translation with Capacity and Compression

arXiv: https://arxiv.org/abs/1808.09943 · HTML: https://mlsanity.com/paper/1808.09943/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2018.

## Built on by (most cited first, 28 of 28)

- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer](https://mlsanity.com/paper/2305.14280/) (2023)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Extended Parallel Corpus for Amharic-English Machine Translation](https://mlsanity.com/paper/2104.03543/) (2021)
- [Improving Arabic Diacritization by Learning to Diacritize and Translate](https://mlsanity.com/paper/2109.14150/) (2021)
- [Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching](https://mlsanity.com/paper/2111.01231/) (2021)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT](https://mlsanity.com/paper/2306.01393/) (2023)
- [Tokenization for Molecular Foundation Models](https://mlsanity.com/paper/2409.15370/) (2024)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on Chinese and English](https://mlsanity.com/paper/2209.02145/) (2022)
- [Improving Language and Modality Transfer in Translation by Character-level Modeling](https://mlsanity.com/paper/2505.24561/) (2025)
- [The boundaries of meaning: a case study in neural machine translation](https://mlsanity.com/paper/2210.00613/) (2022)
- [Investigating Lexical Sharing in Multilingual Machine Translation for Indian Languages](https://mlsanity.com/paper/2305.03207/) (2023)
- [SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation](https://mlsanity.com/paper/2307.16400/) (2023)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://mlsanity.com/paper/2601.11002/) (2026)
- [Does Character-level Information Always Improve DRS-based Semantic Parsing?](https://mlsanity.com/paper/2306.02302/) (2023)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://mlsanity.com/paper/2509.21801/) (2025)
- [Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance](https://mlsanity.com/paper/2511.03383/) (2025)

## Abstract (arXiv)

Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for character-level NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.