# Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation

arXiv: https://arxiv.org/abs/1904.04079 · HTML: https://mlsanity.com/paper/1904.04079/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 5 of 5)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training](https://mlsanity.com/paper/2305.02031/) (2023)
- [Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation](https://mlsanity.com/paper/2109.06308/) (2021)
- [Soft Alignment Objectives for Robust Adaptation of Language Generation](https://mlsanity.com/paper/2211.16550/) (2022)

## Abstract (arXiv)

Despite some empirical success at correcting exposure bias in machine translation, scheduled sampling algorithms suffer from a major drawback: they incorrectly assume that words in the reference translations and in sampled sequences are aligned at each time step. Our new differentiable sampling algorithm addresses this issue by optimizing the probability that the reference can be aligned with the sampled output, based on a soft alignment predicted by the model itself. As a result, the output distribution at each time step is evaluated with respect to the whole predicted sequence. Experiments on IWSLT translation tasks show that our approach improves BLEU compared to maximum likelihood and scheduled sampling baselines. In addition, our approach is simpler to train with no need for sampling schedule and yields models that achieve larger improvements with smaller beam sizes.