mlsanityData to 7 Oct 2026

Prompting PaLM for Translation: Assessing Strategies and Performance

arXiv 2211.09102 ↗First posted 16 Nov 2022cs.CLcited by 144

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM's MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM's MT output which reveals some interesting properties and prospects for future work.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 40 of its 94 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

40 of 94 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
28 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  9. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  10. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  11. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  12. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  13. 2020Nearest Neighbor Machine Translationcited by 209
  14. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  15. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  16. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  17. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  18. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  19. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 52
  20. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  21. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  22. 2022Using natural language prompts for machine translationcited by 27
  23. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  24. 2022Learning to Transfer Prompts for Text Generationcited by 22
  25. 2022MetaPrompting: Learning to Learn Better Promptscited by 19
  26. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  27. 2022Probing via Promptingcited by 7
  28. 2022Prompting for a conversation: How to control a dialog model?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 55; 2024: 52; 2025: 21; 2026: 16.

144 papers.

Sort:

By area:

  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  4. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  5. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  6. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  7. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  8. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  9. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  11. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  12. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
38 more
  1. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  2. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  3. 2023Do Multilingual Language Models Think Better in English?cited by 87
  4. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  5. 2023Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translationcited by 74
  6. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  7. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  8. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  9. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  10. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  11. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  12. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  13. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  14. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  15. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  16. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  17. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  18. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  19. 2023Do GPTs Produce Less Literal Translations?cited by 30
  20. 2023HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysiscited by 30
  21. 2024Teaching Large Language Models an Unseen Language on the Flycited by 29
  22. 2023Iterative Translation Refinement with Large Language Modelscited by 27
  23. 2024TasTe: Teaching Large Language Models to Translate through Self-Reflectioncited by 27
  24. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  25. 2023Augmenting Large Language Model Translators via Translation Memoriescited by 21
  26. 2023Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editingcited by 21
  27. 2024In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translationcited by 21
  28. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  29. 2024ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysiscited by 18
  30. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  31. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  32. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  33. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  34. 2023Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interactioncited by 16
  35. 2023Chain-of-Dictionary Prompting Elicits Translation in Large Language Modelscited by 16
  36. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 16
  37. 2023Deliberate then Generate: Enhanced Prompting Framework for Text Generationcited by 13
  38. 2024Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructionscited by 13

50 most cited of 144 All 144 (JSON)