mlsanityData to 7 Oct 2026

MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation

arXiv 2411.01474 ↗First posted 3 Nov 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages. This avoids out-of-vocabulary risk in multilingual translation and enables broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Mixture of Contextualization Experts (MoCE), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and allows models to search for better contextualization combinations. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. Our code is available at https://github.com/ictnlp/MoCE.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
25 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  2. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  3. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  4. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  5. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  6. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  9. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  10. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  11. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  12. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  13. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  14. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  15. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  16. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  17. 2020On Long-Tailed Phenomena in Neural Machine Translationcited by 20
  18. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  19. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  20. 2020Neural Machine Translation without Embeddingscited by 10
  21. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  22. 2023Enhancing Neural Machine Translation with Semantic Unitscited by 2
  23. 2023Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approachcited by 2
  24. 2024Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Featurescited by 2
  25. 2024Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translationcited by 1

Built on by

Not cited yet.