mlsanityData to 7 Oct 2026

Vocabulary Learning via Optimal Transport for Neural Machine Translation

arXiv 2012.15671 ↗First posted 31 Dec 2020cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

The choice of token vocabulary affects the performance of machine translation. This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of the role of vocabulary from the perspective of information theory. Motivated by this, we formulate the quest of vocabularization -- finding the best token dictionary with a proper size -- as an optimal transport (OT) problem. We propose VOLT, a simple and efficient solution without trial training. Empirical results show that VOLT outperforms widely-used vocabularies in diverse scenarios, including WMT-14 English-German and TED's 52 translation directions. For example, VOLT achieves almost 70% vocabulary size reduction and 0.5 BLEU gain on English-German translation. Also, compared to BPE-search, VOLT reduces the search time from 384 GPU hours to 30 GPU hours on English-German translation. Codes are available at https://github.com/Jingjing-NLP/VOLT .

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 19; 2023: 11; 2024: 17; 2025: 8.

58 papers.

Sort:

By area:

  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  2. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  3. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  4. 2024LLMs are Good Sign Language Translatorscited by 76
  5. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  6. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  7. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
  8. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  9. 2022Rethinking and Refining the Distinct Metriccited by 27
  10. 2022Revisiting Sliced Wasserstein on Images: From Vectorization to Convolutioncited by 25
  11. 2022Hierarchical Sliced Wasserstein Distancecited by 23
  12. 2022Partially Does It: Towards Scene-Level FG-SBIR with Partial Inputcited by 20
38 more
  1. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  2. 2022Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translationcited by 18
  3. 2023Wasserstein-Fisher-Rao Embedding: Logical Query Embeddings with Local Comparison and Global Transportcited by 16
  4. 2024Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removalcited by 16
  5. 2021The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21cited by 14
  6. 2022Toward Interpretable Semantic Textual Similarity via Optimal Transport-based Contrastive Sentence Learningcited by 13
  7. 2024Lexically Grounded Subword Segmentationcited by 13
  8. 2025Protein Structure Tokenization: Benchmarking and New Recipecited by 13
  9. 2025DistDF: Time-Series Forecasting Needs Joint-Distribution Wasserstein Alignmentcited by 13
  10. 2024Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generationcited by 11
  11. 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
  12. 2022OTExtSum: Extractive Text Summarisation with Optimal Transportcited by 8
  13. 2023T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classificationcited by 8
  14. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  15. 2024Language Generation with Strictly Proper Scoring Rulescited by 8
  16. 2022Learning Multiscale Transformer Models for Sequence Generationcited by 7
  17. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7
  18. 2022Subword-Delimited Downsampling for Better Character-Level Translationcited by 7
  19. 2022Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Wordscited by 6
  20. 2024Hierarchical Hybrid Sliced Wasserstein: A Scalable Metric for Heterogeneous Joint Distributionscited by 6
  21. 2025FreeMesh: Boosting Mesh Generation with Coordinates Mergingcited by 6
  22. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  23. 2024LBPE: Long-token-first Tokenization to Improve Large Language Modelscited by 5
  24. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generationcited by 5
  25. 2024Computing Approximate Graph Edit Distance via Optimal Transportcited by 5
  26. 2025A Partition Cover Approach to Tokenizationcited by 5
  27. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  28. 2024Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansioncited by 4
  29. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  30. 2022GeONet: a neural operator for learning the Wasserstein geodesiccited by 3
  31. 2023On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translationcited by 3
  32. 2024Target-Aware Language Modeling via Granular Data Samplingcited by 3
  33. 2023TOT: Topology-Aware Optimal Transport For Multimodal Hate Detectioncited by 2
  34. 2024Infusing clinical knowledge into tokenisers for language modelscited by 2
  35. 2023Extrapolating Multilingual Understanding Models as Multilingual Generatorscited by 1
  36. 2023MobileNMT: Enabling Translation in 15MB and 30mscited by 1
  37. 2023SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translationcited by 1
  38. 2023Compressed online Sinkhorncited by 1

50 most cited of 58 All 58 (JSON)