mlsanityData to 14 Oct 2026

Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Language

arXiv 2103.08052 ↗First posted 14 Mar 2021cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Building effective neural machine translation (NMT) models for very low-resourced and morphologically rich African indigenous languages is an open challenge. Besides the issue of finding available resources for them, a lot of work is put into preprocessing and tokenization. Recent studies have shown that standard tokenization methods do not always adequately deal with the grammatical, diacritical, and tonal properties of some African languages. That, coupled with the extremely low availability of training samples, hinders the production of reliable NMT models. In this paper, using Fon language as a case study, we revisit standard tokenization methods and introduce Word-Expressions-Based (WEB) tokenization, a human-involved super-words tokenization strategy to create a better representative vocabulary for training. Furthermore, we compare our tokenization strategy to others on the Fon-French and French-Fon translation tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

18 of 36 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  9. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  10. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  11. 2020Masakhane -- Machine Translation For Africacited by 49
  12. 2019Using Monolingual Data in Neural Machine Translation: a Systematic Studycited by 21
6 more
  1. 2017The University of Edinburgh's Neural MT Systems for WMT17cited by 16
  2. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  3. 2020On Optimal Transformer Depth for Low-Resource Language Translationcited by 9
  4. 2018Towards Neural Machine Translation for African Languagescited by 7
  5. 2019Overcoming the Rare Word Problem for Low-Resource Language Pairs in Neural Machine Translationcited by 3
  6. 2020FFR V1.0: Fon-French Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 0; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  3. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  4. 2022MMTAfrica: Multilingual Machine Translation for African Languagescited by 11
  5. 2025Towards Open-Ended Discovery for Low-Resource NLPcited by 1
  6. 2026A Survey of Text and Speech Resources for Hausa and Fongbe: Availability, Quality, and Gaps for NLP Developmentcited by 0