How Important is `Perfect' English for Machine Translation Prompts?
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have achieved top results in recent machine translation evaluations, but they are also known to be sensitive to errors and perturbations in their prompts. We systematically evaluate how both humanly plausible and synthetic errors in user prompts affect LLMs' performance on two related tasks: Machine translation and machine translation evaluation. We provide both a quantitative analysis and qualitative insights into how the models respond to increasing noise in the user prompt. The prompt quality strongly affects the translation performance: With many errors, even a good prompt can underperform a minimal or poor prompt without errors. However, different noise types impact translation quality differently, with character-level and combined noisers degrading performance more than phrasal perturbations. Qualitative analysis reveals that lower prompt quality largely leads to poorer instruction following, rather than directly affecting translation quality itself. Further, LLMs can still translate in scenarios with overwhelming random noise that would make the prompt illegible to humans.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 13 Jul 2025.
Builds on
23 of 38 references.
References · 23
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
- 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
- 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
- 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
- 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
11 more
- 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
- 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
- 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
- 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 59
- 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
- 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
- 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
- 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
- 2024Did Translation Models Get More Robust Without Anyone Even Noticing?cited by 9
- 2024Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalizationcited by 7
- 2024We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Textcited by 6
Built on by
Citing papers per year: 2025: 2.
2 papers.