# Generating Diverse Translation by Manipulating Multi-Head Attention

arXiv: https://arxiv.org/abs/1911.09333 · HTML: https://mlsanity.com/paper/1911.09333/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Mixup Decoding for Diverse Machine Translation](https://mlsanity.com/paper/2109.03402/) (2021)
- [To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translation](https://mlsanity.com/paper/2401.01419/) (2024)
- [WeTS: A Benchmark for Translation Suggestion](https://mlsanity.com/paper/2110.05151/) (2021)
- [Zero-shot Domain Adaptation for Neural Machine Translation with Retrieved Phrase-level Prompts](https://mlsanity.com/paper/2209.11409/) (2022)
- [Controlling Styles in Neural Machine Translation with Activation Prompt](https://mlsanity.com/paper/2212.08909/) (2022)
- [Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention](https://mlsanity.com/paper/2211.07164/) (2022)
- [Generating Diverse Translation with Perturbed kNN-MT](https://mlsanity.com/paper/2402.09344/) (2024)
- [EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation](https://mlsanity.com/paper/2203.02180/) (2022)
- [Only 5% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation](https://mlsanity.com/paper/2309.14174/) (2023)
- [Diverse Sign Language Translation](https://mlsanity.com/paper/2410.19586/) (2024)
- [DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model](https://mlsanity.com/paper/2411.17248/) (2024)
- [Iterative Feature Space Optimization through Incremental Adaptive Evaluation](https://mlsanity.com/paper/2501.14889/) (2025)
- [A Static and Dynamic Attention Framework for Multi Turn Dialogue Generation](https://mlsanity.com/paper/2410.20766/) (2024)

## Abstract (arXiv)

Transformer model has been widely used on machine translation tasks and obtained state-of-the-art results. In this paper, we report an interesting phenomenon in its encoder-decoder multi-head attention: different attention heads of the final decoder layer align to different word translation candidates. We empirically verify this discovery and propose a method to generate diverse translations by manipulating heads. Furthermore, we make use of these diverse translations with the back-translation technique for better data augmentation. Experiment results show that our method generates diverse translations without severe drop in translation quality. Experiments also show that back-translation with these diverse translations could bring significant improvement on performance on translation tasks. An auxiliary experiment of conversation response generation task proves the effect of diversity as well.