# Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation

arXiv: https://arxiv.org/abs/2109.06253 · HTML: https://mlsanity.com/paper/2109.06253/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [Correcting Length Bias in Neural Machine Translation](https://mlsanity.com/paper/1808.10006/) (2018)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Calibration of Encoder Decoder Models for Neural Machine Translation](https://mlsanity.com/paper/1903.00802/) (2019)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation](https://mlsanity.com/paper/1905.10453/) (2019)
- [Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation](https://mlsanity.com/paper/1808.09582/) (2018)
- [Length bias in Encoder Decoder Models and a Case for Global Conditioning](https://mlsanity.com/paper/1606.03402/) (2016)
- [Robust Beam Search for Encoder-Decoder Attention Based Speech Recognition without Length Bias](https://mlsanity.com/paper/2005.09265/) (2020)

## Built on by (most cited first, 4 of 4)

- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility](https://mlsanity.com/paper/2605.01357/) (2026)
- [Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognition](https://mlsanity.com/paper/2210.15876/) (2022)
- [LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation](https://mlsanity.com/paper/2607.04270/) (2026)

## Abstract (arXiv)

Neural Machine Translation (NMT) is known to suffer from a beam-search problem: after a certain point, increasing beam size causes an overall drop in translation quality. This effect is especially pronounced for long sentences. While much work was done analyzing this phenomenon, primarily for autoregressive NMT models, there is still no consensus on its underlying cause. In this work, we analyze errors that cause major quality degradation with large beams in NMT and Automatic Speech Recognition (ASR). We show that a factor that strongly contributes to the quality degradation with large beams is textitdataset length-bias - textitNMT datasets are strongly biased towards short sentences. To mitigate this issue, we propose a new data augmentation technique -- textitMulti-Sentence Resampling (MSR). This technique extends the training examples by concatenating several sentences from the original dataset to make a long training example. We demonstrate that MSR significantly reduces degradation with growing beam size and improves final translation quality on the IWSTL15 En-Vi, IWSTL17 En-Fr, and WMT14 En-De datasets.