# Do GPTs Produce Less Literal Translations?

arXiv: https://arxiv.org/abs/2305.16806 · HTML: https://mlsanity.com/paper/2305.16806/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability](https://mlsanity.com/paper/2305.10266/) (2023)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [EPIE Dataset: A Corpus For Possible Idiomatic Expressions](https://mlsanity.com/paper/2006.09479/) (2020)
- [SALTED: A Framework for SAlient Long-Tail Translation Error Detection](https://mlsanity.com/paper/2205.09988/) (2022)

## Built on by (most cited first, 30 of 30)

- [Speech Translation with Large Language Models: An Industrial Practice](https://mlsanity.com/paper/2312.13585/) (2023)
- [Iterative Translation Refinement with Large Language Models](https://mlsanity.com/paper/2306.03856/) (2023)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities](https://mlsanity.com/paper/2405.20089/) (2024)
- [Lost in Literalism: How Supervised Training Shapes Translationese in LLMs](https://mlsanity.com/paper/2503.04369/) (2025)
- [Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding](https://mlsanity.com/paper/2311.02851/) (2023)
- [To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translation](https://mlsanity.com/paper/2401.01419/) (2024)
- [Creative and Context-Aware Translation of East Asian Idioms with GPT-4](https://mlsanity.com/paper/2410.00988/) (2024)
- [Towards Effective Disambiguation for Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.11668/) (2023)
- [SCALE: Synergized Collaboration of Asymmetric Language Translation Engines](https://mlsanity.com/paper/2309.17061/) (2023)
- [That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?](https://mlsanity.com/paper/2310.14610/) (2023)
- [Aligning Translation-Specific Understanding to General Understanding in Large Language Models](https://mlsanity.com/paper/2401.05072/) (2024)
- [Extending CREAMT: Leveraging Large Language Models for Literary Translation Post-Editing](https://mlsanity.com/paper/2504.03045/) (2025)
- [Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedish](https://mlsanity.com/paper/2602.03484/) (2026)
- [Dissecting In-Context Learning of Translations in GPTs](https://mlsanity.com/paper/2310.15987/) (2023)
- [Exploring In-context Example Generation for Machine Translation](https://mlsanity.com/paper/2506.00507/) (2025)
- [Killing Two Flies with One Stone: An Attempt to Break LLMs Using English->Icelandic Idioms and Proper Names](https://mlsanity.com/paper/2410.03394/) (2024)
- [Graph-Assisted Culturally Adaptable Idiomatic Translation for Indic Languages](https://mlsanity.com/paper/2505.21937/) (2025)
- [Distinguishing Translations by Human, NMT, and ChatGPT: A Linguistic and Statistical Approach](https://mlsanity.com/paper/2312.10750/) (2023)
- [Prediction of Translation Techniques for the Translation Process](https://mlsanity.com/paper/2403.14454/) (2024)
- [Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation](https://mlsanity.com/paper/2406.07970/) (2024)
- [Large Language Models for Persian  leftrightarrow  English Idiom Translation](https://mlsanity.com/paper/2412.09993/) (2024)
- [Missing the human touch? A computational stylometry analysis of GPT-4 translations of online Chinese literature](https://mlsanity.com/paper/2506.13013/) (2025)
- [ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation](https://mlsanity.com/paper/2509.22768/) (2025)
- [Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decoding](https://mlsanity.com/paper/2510.06866/) (2025)
- [LLMs for Low-Resource Dialect Translation Using Context-Aware Prompting: A Case Study on Sylheti](https://mlsanity.com/paper/2511.21761/) (2025)
- [A Dataset for Probing Translationese Preferences in English-to-Swedish Translation](https://mlsanity.com/paper/2603.08450/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Testing the Deliteralization Hypothesis in Human and Machine Translation](https://mlsanity.com/paper/2605.25686/) (2026)
- [Emotion Profiling in LLM-Based Literary Translation: Systematic Shifts Across MT and Post-Editing](https://mlsanity.com/paper/2606.10113/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations differ qualitatively from the translations generated by standard Neural Machine Translation (NMT) models. In this work, we investigate these differences in terms of the literalness of translations produced by the two systems. Using literalness measures involving word alignment and monotonicity, we find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on MT quality metrics. We demonstrate that this finding is borne out in human evaluations as well. We then show that these differences are especially pronounced when translating sentences that contain idiomatic expressions.