# Controllable Dual Skew Divergence Loss for Neural Machine Translation

arXiv: https://arxiv.org/abs/1908.08399 · HTML: https://mlsanity.com/paper/1908.08399/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 2 of 2)

- [Mixed Cross Entropy Loss for Neural Machine Translation](https://mlsanity.com/paper/2106.15880/) (2021)
- [α-Geodesical Skew Divergence](https://mlsanity.com/paper/2103.17060/) (2021)

## Abstract (arXiv)

In sequence prediction tasks like neural machine translation, training with cross-entropy loss often leads to models that overgeneralize and plunge into local optima. In this paper, we propose an extended loss function called dual skew divergence (DSD) that integrates two symmetric terms on KL divergences with a balanced weight. We empirically discovered that such a balanced weight plays a crucial role in applying the proposed DSD loss into deep models. Thus we eventually develop a controllable DSD loss for general-purpose scenarios. Our experiments indicate that switching to the DSD loss after the convergence of ML training helps models escape local optima and stimulates stable performance improvements. Our evaluations on the WMT 2014 English-German and English-French translation tasks demonstrate that the proposed loss as a general and convenient mean for NMT training indeed brings performance improvement in comparison to strong baselines.