# Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

arXiv: https://arxiv.org/abs/2205.15301 · HTML: https://mlsanity.com/paper/2205.15301/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [Examining the Tip of the Iceberg: A Data Set for Idiom Translation](https://mlsanity.com/paper/1802.04681/) (2018)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition](https://mlsanity.com/paper/1902.10618/) (2019)
- [An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation](https://mlsanity.com/paper/1810.07595/) (2018)
- [On Compositionality in Neural Machine Translation](https://mlsanity.com/paper/1911.01497/) (2019)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method](https://mlsanity.com/paper/1711.07646/) (2017)
- [On the Sub-Layer Functionalities of Transformer Decoder](https://mlsanity.com/paper/2010.02648/) (2020)
- [Encoders Help You Disambiguate Word Senses in Neural Machine Translation](https://mlsanity.com/paper/1908.11771/) (2019)
- [Linguistic evaluation of German-English Machine Translation using a Test Suite](https://mlsanity.com/paper/1910.07457/) (2019)

## Built on by (most cited first, 28 of 28)

- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models](https://mlsanity.com/paper/2308.13961/) (2023)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks](https://mlsanity.com/paper/2210.12929/) (2022)
- [Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://mlsanity.com/paper/2310.07081/) (2023)
- [Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models](https://mlsanity.com/paper/2210.03575/) (2022)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://mlsanity.com/paper/2210.04545/) (2022)
- [Hierarchical Phrase-based Sequence-to-Sequence Learning](https://mlsanity.com/paper/2211.07906/) (2022)
- [From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation](https://mlsanity.com/paper/2408.14825/) (2024)
- [Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations](https://mlsanity.com/paper/2305.14599/) (2023)
- [PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese](https://mlsanity.com/paper/2404.05333/) (2024)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)
- [Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedish](https://mlsanity.com/paper/2602.03484/) (2026)
- [Automating Behavioral Testing in Machine Translation](https://mlsanity.com/paper/2309.02553/) (2023)
- [Does the "most sinfully decadent cake ever" taste good? Answering Yes/No Questions from Figurative Contexts](https://mlsanity.com/paper/2309.13748/) (2023)
- [GLoT: A Novel Gated-Logarithmic Transformer for Efficient Sign Language Translation](https://mlsanity.com/paper/2502.12223/) (2025)
- [Evaluating LLMs on Chinese Idiom Translation](https://mlsanity.com/paper/2508.10421/) (2025)
- [Predicting Human Translation Difficulty with Neural Machine Translation](https://mlsanity.com/paper/2312.11852/) (2023)
- [The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relations](https://mlsanity.com/paper/2404.08661/) (2024)
- [Compositional Generalization with Grounded Language Models](https://mlsanity.com/paper/2406.04989/) (2024)
- [Anatomy of an Idiom: Tracing Non-Compositionality in Language Models](https://mlsanity.com/paper/2511.16467/) (2025)
- [Evaluating Communicative Success in Machine-Translated Conversation](https://mlsanity.com/paper/2609.19885/) (2026)

## Abstract (arXiv)

Unlike literal expressions, idioms' meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation - i.e. identifies the expression as idiomatic - the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms' parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder's cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer's tendency to process idioms as compositional expressions contributes to literal translations of idioms.