# Killing Two Flies with One Stone: An Attempt to Break LLMs Using English->Icelandic Idioms and Proper Names

arXiv: https://arxiv.org/abs/2410.03394 · HTML: https://mlsanity.com/paper/2410.03394/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 6 of 6)

- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Examining the Tip of the Iceberg: A Data Set for Idiom Translation](https://mlsanity.com/paper/1802.04681/) (2018)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities](https://mlsanity.com/paper/2405.20089/) (2024)
- [Potential Idiomatic Expression (PIE)-English: Corpus for Classes of Idioms](https://mlsanity.com/paper/2105.03280/) (2021)
- [Nefnir: A high accuracy lemmatizer for Icelandic](https://mlsanity.com/paper/1907.11907/) (2019)

## Built on by (most cited first, 1 of 1)

- [Generating Difficult-to-Translate Texts](https://mlsanity.com/paper/2509.26592/) (2025)

## Abstract (arXiv)

This paper presents the submission of the 'Arni Magn'usson Institute's team to the WMT24 test suite subtask, focusing on idiomatic expressions and proper names for the English->Icelandic translation direction. Intuitively and empirically, idioms and proper names are known to be a significant challenge for modern translation models. We create two different test suites. The first evaluates the competency of MT systems in translating common English idiomatic expressions, as well as testing whether systems can distinguish between those expressions and the same phrases when used in a literal context. The second test suite consists of place names that should be translated into their Icelandic exonyms (and correctly inflected) and pairs of Icelandic names that share a surface form between the male and female variants, so that incorrect translations impact meaning as well as readability. The scores reported are relatively low, especially for idiomatic expressions and place names, and indicate considerable room for improvement.