mlsanityData to 7 Oct 2026

An Empirical Study of Translation Hypothesis Ensembling with Large Language Models

arXiv 2310.11430 ↗First posted 17 Oct 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. In this paper, we investigate how hypothesis ensembling can improve the quality of the generated text for the specific problem of LLM-based machine translation. We experiment with several techniques for ensembling hypotheses produced by LLMs such as ChatGPT, LLaMA, and Alpaca. We provide a comprehensive study along multiple dimensions, including the method to generate hypotheses (multiple prompts, temperature-based sampling, and beam search) and the strategy to produce the final translation (instruction-based, quality-based reranking, and minimum Bayes risk (MBR) decoding). Our results show that MBR decoding is a very effective method, that translation quality can be improved using a small number of samples, and that instruction tuning has a strong impact on the relation between the diversity of the hypotheses and the sampling temperature.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

46 of 69 references.

References · 46

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
34 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  10. 2016FastText.zip: Compressing text classification modelscited by 331
  11. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  12. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  13. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  14. 2022Can large language models reason about medical questions?cited by 194
  15. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  16. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  17. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  18. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  19. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  20. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  21. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  22. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  23. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  24. 2022Recitation-Augmented Language Modelscited by 65
  25. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  26. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  27. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  28. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  29. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  30. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  31. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  32. 2023Iterative Translation Refinement with Large Language Modelscited by 27
  33. 2016A Character-Level Decoder without Explicit Segmentation for Neural Machine Translationcited by 21
  34. 2022Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 5; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  2. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  3. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  4. 2024Cool-Fusion: Fuse Large Language Models without Trainingcited by 12
  5. 2023Model-Based Minimum Bayes Risk Decoding for Text Generationcited by 10
  6. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  7. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  8. 2025Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translationcited by 6
  9. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  10. 2024MQM-Chat: Multidimensional Quality Metrics for Chat Translationcited by 3
  11. 2024Reranking Laws for Language Generation: A Communication-Theoretic Perspectivecited by 3
  12. 2024A Context-aware Framework for Translation-mediated Conversationscited by 2
4 more
  1. 2025DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Modelcited by 0
  2. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 0
  3. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  4. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0