# Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation

arXiv: https://arxiv.org/abs/2203.13528 · HTML: https://mlsanity.com/paper/2203.13528/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [Joint Optimization of Tokenization and Downstream Model](https://mlsanity.com/paper/2105.12410/) (2021)
- [Single Model Ensemble using Pseudo-Tags and Distinct Vectors](https://mlsanity.com/paper/2005.00879/) (2020)

## Built on by (most cited first, 2 of 2)

- [MaxMatch-Dropout: Subword Regularization for WordPiece](https://mlsanity.com/paper/2209.04126/) (2022)
- [SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization](https://mlsanity.com/paper/2409.06216/) (2024)

## Abstract (arXiv)

Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we use multiple segmentations in the training process but use only one segmentation in the inference. In this study, we propose an inference strategy to address this discrepancy. The proposed strategy approximates the marginalized likelihood by using multiple segmentations including the most plausible segmentation and several sampled segmentations. Because the proposed strategy aggregates predictions from several segmentations, we can regard it as a single model ensemble that does not require any additional cost for training. Experimental results show that the proposed strategy improves the performance of models trained with subword regularization in low-resource machine translation tasks.