# Encoders Help You Disambiguate Word Senses in Neural Machine Translation

arXiv: https://arxiv.org/abs/1908.11771 · HTML: https://mlsanity.com/paper/1908.11771/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2019.

## Built on by (most cited first, 6 of 6)

- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [Towards Effective Disambiguation for Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.11668/) (2023)
- [Code-Switching with Word Senses for Pretraining in Neural Machine Translation](https://mlsanity.com/paper/2310.14050/) (2023)
- [Predicting Human Translation Difficulty with Neural Machine Translation](https://mlsanity.com/paper/2312.11852/) (2023)
- [Learning Homographic Disambiguation Representation for Neural Machine Translation](https://mlsanity.com/paper/2304.05860/) (2023)

## Abstract (arXiv)

Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. Decoders could provide further relevant information for disambiguation. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. Note that this is a revised version. The content related to decoder hidden states has been updated.