# Context-Dependent Word Representation for Neural Machine Translation

arXiv: https://arxiv.org/abs/1607.00578 · HTML: https://mlsanity.com/paper/1607.00578/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2016.

## Built on by (most cited first, 7 of 7)

- [Towards Effective Disambiguation for Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.11668/) (2023)
- [Code-Switching with Word Senses for Pretraining in Neural Machine Translation](https://mlsanity.com/paper/2310.14050/) (2023)
- [Towards Network Traffic Monitoring Using Deep Transfer Learning](https://mlsanity.com/paper/2101.00731/) (2021)
- [CRAT: A Multi-Agent Framework for Causality-Enhanced Reflective and Retrieval-Augmented Translation with Large Language Models](https://mlsanity.com/paper/2410.21067/) (2024)
- [Word2rate: training and evaluating multiple word embeddings as statistical transitions](https://mlsanity.com/paper/2104.08173/) (2021)
- [Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network](https://mlsanity.com/paper/2111.11707/) (2021)
- [An Evaluation Dataset for Legal Word Embedding: A Case Study On Chinese Codex](https://mlsanity.com/paper/2203.15173/) (2022)

## Abstract (arXiv)

We first observe a potential weakness of continuous vector representations of symbols in neural machine translation. That is, the continuous vector representation, or a word embedding vector, of a symbol encodes multiple dimensions of similarity, equivalent to encoding more than one meaning of the word. This has the consequence that the encoder and decoder recurrent networks in neural machine translation need to spend substantial amount of their capacity in disambiguating source and target words based on the context which is defined by a source sentence. Based on this observation, in this paper we propose to contextualize the word embedding vectors using a nonlinear bag-of-words representation of the source sentence. Additionally, we propose to represent special tokens (such as numbers, proper nouns and acronyms) with typed symbols to facilitate translating those words that are not well-suited to be translated via continuous vectors. The experiments on En-Fr and En-De reveal that the proposed approaches of contextualization and symbolization improves the translation quality of neural machine translation systems significantly.