# SOTASTREAM: A Streaming Approach to Machine Translation Training

arXiv: https://arxiv.org/abs/2308.07489 · HTML: https://mlsanity.com/paper/2308.07489/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [AugLy: Data Augmentations for Robustness](https://mlsanity.com/paper/2201.06494/) (2022)
- [Many-to-English Machine Translation Tools, Data, and Pretrained Models](https://mlsanity.com/paper/2104.00290/) (2021)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [Marian: Cost-effective High-Quality Neural Machine Translation in C++](https://mlsanity.com/paper/1805.12096/) (2018)
- [Facilitating Terminology Translation with Target Lemma Annotations](https://mlsanity.com/paper/2101.10035/) (2021)
- [Findings of the First Shared Task on Machine Translation Robustness](https://mlsanity.com/paper/1906.11943/) (2019)
- [Guided Alignment Training for Topic-Aware Neural Machine Translation](https://mlsanity.com/paper/1607.01628/) (2016)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [Adam Mickiewicz University at WMT 2022: NER-Assisted and Quality-Aware Neural Machine Translation](https://mlsanity.com/paper/2209.02962/) (2022)

## Built on by (most cited first, 3 of 3)

- [EMMeTT: Efficient Multimodal Machine Translation Training](https://mlsanity.com/paper/2409.13523/) (2024)
- [Token-level Ensembling of Models with Different Vocabularies](https://mlsanity.com/paper/2502.21265/) (2025)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)

## Abstract (arXiv)

Many machine translation toolkits make use of a data preparation step wherein raw data is transformed into a tensor format that can be used directly by the trainer. This preparation step is increasingly at odds with modern research and development practices because this process produces a static, unchangeable version of the training data, making common training-time needs difficult (e.g., subword sampling), time-consuming (preprocessing with large data can take days), expensive (e.g., disk space), and cumbersome (managing experiment combinatorics). We propose an alternative approach that separates the generation of data from the consumption of that data. In this approach, there is no separate pre-processing step; data generation produces an infinite stream of permutations of the raw training data, which the trainer tensorizes and batches as it is consumed. Additionally, this data stream can be manipulated by a set of user-definable operators that provide on-the-fly modifications, such as data normalization, augmentation or filtering. We release an open-source toolkit, SOTASTREAM, that implements this approach: https://github.com/marian-nmt/sotastream. We show that it cuts training time, adds flexibility, reduces experiment management complexity, and reduces disk space, all without affecting the accuracy of the trained models.