# Enhancing Neural Machine Translation with Semantic Units

arXiv: https://arxiv.org/abs/2310.11360 · HTML: https://mlsanity.com/paper/2310.11360/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [Graph Convolutional Encoders for Syntax-aware Neural Machine Translation](https://mlsanity.com/paper/1704.04675/) (2017)
- [BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models](https://mlsanity.com/paper/2306.10968/) (2023)
- [UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining](https://mlsanity.com/paper/2202.13469/) (2022)
- [Tree-to-Sequence Attentional Neural Machine Translation](https://mlsanity.com/paper/1603.06075/) (2016)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Multi-Granularity Self-Attention for Neural Machine Translation](https://mlsanity.com/paper/1909.02222/) (2019)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [Semantics-aware Attention Improves Neural Machine Translation](https://mlsanity.com/paper/2110.06920/) (2021)
- [Learning Source Phrase Representations for Neural Machine Translation](https://mlsanity.com/paper/2006.14405/) (2020)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [Categorizing Semantic Representations for Neural Machine Translation](https://mlsanity.com/paper/2210.06709/) (2022)

## Built on by (most cited first, 2 of 2)

- [Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation](https://mlsanity.com/paper/2405.19290/) (2024)
- [MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation](https://mlsanity.com/paper/2411.01474/) (2024)

## Abstract (arXiv)

Conventional neural machine translation (NMT) models typically use subwords and words as the basic units for model input and comprehension. However, complete words and phrases composed of several tokens are often the fundamental units for expressing semantics, referred to as semantic units. To address this issue, we propose a method Semantic Units for Machine Translation (SU4MT) which models the integral meanings of semantic units within a sentence, and then leverages them to provide a new perspective for understanding the sentence. Specifically, we first propose Word Pair Encoding (WPE), a phrase extraction method to help identify the boundaries of semantic units. Next, we design an Attentive Semantic Fusion (ASF) layer to integrate the semantics of multiple subwords into a single vector: the semantic unit representation. Lastly, the semantic-unit-level sentence representation is concatenated to the token-level one, and they are combined as the input of encoder. Experimental results demonstrate that our method effectively models and leverages semantic-unit-level information and outperforms the strong baselines. The code is available at https://github.com/ictnlp/SU4MT.