mlsanityData to 7 Oct 2026

Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

arXiv 2301.08745 ↗First posted 20 Jan 2023cs.CLcited by 172

Not on the map. Why?

Abstract (arXiv)

This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named pivot~prompting for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 20 Jan 2023.

Builds on

14 of 28 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  7. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  8. 2022In-context Examples Selection for Machine Translationcited by 172
  9. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  10. 2021Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translationcited by 17
  11. 2022Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languagescited by 13
  12. 2022Understanding and Mitigating the Uncertainty in Zero-Shot Translationcited by 9
2 more
  1. 2022Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Taskcited by 8
  2. 2022ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 69; 2024: 64; 2025: 28; 2026: 11.

172 papers.

Sort:

By area:

  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  2. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  5. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  6. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  7. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  9. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  10. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  11. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  12. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
38 more
  1. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  2. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  3. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  4. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  5. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  6. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  7. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  8. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  9. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  10. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  11. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  12. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  13. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  14. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  15. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  16. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  17. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  18. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  19. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  20. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46
  21. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  22. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
  23. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  24. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 38
  25. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  26. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  27. 2024LLMs are Good Action Recognizerscited by 36
  28. 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
  29. 2023When do you need Chain-of-Thought Prompting for ChatGPT?cited by 34
  30. 2023A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understandingcited by 31
  31. 2023The Perils & Promises of Fact-checking with Large Language Modelscited by 31
  32. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  33. 2023Iterative Translation Refinement with Large Language Modelscited by 27
  34. 2024DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memorycited by 27
  35. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  36. 2023Towards LLM-driven Dialogue State Trackingcited by 25
  37. 2023Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Studycited by 24
  38. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24

50 most cited of 172 All 172 (JSON)