# That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?

arXiv: https://arxiv.org/abs/2310.14610 · HTML: https://mlsanity.com/paper/2310.14610/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation](https://mlsanity.com/paper/2210.07626/) (2022)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Context-Aware Monolingual Repair for Neural Machine Translation](https://mlsanity.com/paper/1909.01383/) (2019)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [Examining the Tip of the Iceberg: A Data Set for Idiom Translation](https://mlsanity.com/paper/1802.04681/) (2018)
- [Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand](https://mlsanity.com/paper/2112.04139/) (2021)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [Automatically Identifying Gender Issues in Machine Translation using Perturbations](https://mlsanity.com/paper/2004.14065/) (2020)
- [EPIE Dataset: A Corpus For Possible Idiomatic Expressions](https://mlsanity.com/paper/2006.09479/) (2020)
- [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021)
- [Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA](https://mlsanity.com/paper/2211.07516/) (2022)
- [Pick a Fight or Bite your Tongue: Investigation of Gender Differences in Idiomatic Language Usage](https://mlsanity.com/paper/2011.00335/) (2020)
- [Paying Attention to Multi-Word Expressions in Neural Machine Translation](https://mlsanity.com/paper/1710.06313/) (2017)

## Built on by (most cited first, 3 of 3)

- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [Memorization or Reasoning? Exploring the Idiom Understanding of LLMs](https://mlsanity.com/paper/2505.16216/) (2025)
- [Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark](https://mlsanity.com/paper/2608.04670/) (2026)

## Abstract (arXiv)

The translation of ambiguous text presents a challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible. While prior work has studied ambiguities that result from different grammatical features of the source and target language, we study semantic ambiguities that exist in the source (English in this work) itself. In particular, we focus on idioms that are open to both literal and figurative interpretations (e.g., goose egg), and collect TIDE, a dataset of 512 pairs of English sentences containing idioms with disambiguating context such that one is literal (it laid a goose egg) and another is figurative (they scored a goose egg, as in a score of zero). In experiments, we compare MT-specific models and language models for (i) their preference when given an ambiguous subsentence, (ii) their sensitivity to disambiguating context, and (iii) the performance disparity between figurative and literal source sentences. We find that current MT models consistently translate English idioms literally, even when the context suggests a figurative interpretation. On the other hand, LMs are far more context-aware, although there remain disparities across target languages. Our findings underline the potential of LMs as a strong backbone for context-aware translation.