# Revisiting Syllables in Language Modelling and their Application on Low-Resource Machine Translation

arXiv: https://arxiv.org/abs/2210.02509 · HTML: https://mlsanity.com/paper/2210.02509/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [Challenges of language technologies for the indigenous languages of the Americas](https://mlsanity.com/paper/1806.04291/) (2018)
- [Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English](https://mlsanity.com/paper/1707.09879/) (2017)
- [Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2201.03110/) (2022)
- [Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model](https://mlsanity.com/paper/1804.08205/) (2018)
- [Better Character Language Modeling Through Morphology](https://mlsanity.com/paper/1906.01037/) (2019)
- [Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling](https://mlsanity.com/paper/1704.06986/) (2017)
- [Orthographic Syllable as basic unit for SMT between Related Languages](https://mlsanity.com/paper/1610.00634/) (2016)
- [OTEANN: Estimating the Transparency of Orthographies with an Artificial Neural Network](https://mlsanity.com/paper/1912.13321/) (2019)
- [Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones](https://mlsanity.com/paper/1707.06480/) (2017)
- [A Syllable-based Technique for Word Embeddings of Korean Words](https://mlsanity.com/paper/1708.01766/) (2017)
- [Syllable-level Neural Language Model for Agglutinative Language](https://mlsanity.com/paper/1708.05515/) (2017)

## Abstract (arXiv)

Language modelling and machine translation tasks mostly use subword or character inputs, but syllables are seldom used. Syllables provide shorter sequences than characters, require less-specialised extracting rules than morphemes, and their segmentation is not impacted by the corpus size. In this study, we first explore the potential of syllables for open-vocabulary language modelling in 21 languages. We use rule-based syllabification methods for six languages and address the rest with hyphenation, which works as a syllabification proxy. With a comparable perplexity, we show that syllables outperform characters and other subwords. Moreover, we study the importance of syllables on neural machine translation for a non-related and low-resource language-pair (Spanish--Shipibo-Konibo). In pairwise and multilingual systems, syllables outperform unsupervised subwords, and further morphological segmentation methods, when translating into a highly synthetic language with a transparent orthography (Shipibo-Konibo). Finally, we perform some human evaluation, and discuss limitations and opportunities.