mlsanityData to 14 Oct 2026

On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation

arXiv 2601.13729 ↗First posted 20 Jan 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent years, the non-deterministic properties of language models have garnered considerable attention and have shown a significant influence on real-world applications. However, such properties remain under-explored in machine translation (MT), a complex, non-deterministic NLP task. In this study, we systematically evaluate modern MT systems and identify temperature-constrained Non-Deterministic MT (ND-MT) as a distinct phenomenon. Additionally, we demonstrate that ND-MT exhibits significant potential in addressing the multimodality issue that has long challenged MT research and provides higher-quality candidates than Deterministic MT (D-MT) under temperature constraints. However, ND-MT introduces new challenges in evaluating system performance. Specifically, the evaluation framework designed for D-MT fails to yield consistent evaluation results when applied to ND-MT. We further investigate this emerging challenge by evaluating state-of-the-art ND-MT systems using both lexical-based and semantic-based metrics at varying sampling sizes. The results reveal a Buckets Effect across these systems: the ranking of ND-MT systems is dominated by the worst-quality candidate translation, as shown by automatic evaluation metrics. To mitigate this issue, we propose ExpectoSample, a strategy that first identifies reliable metrics and then enables robust ND-MT system selection for real-world.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

36 of 52 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
24 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  5. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  6. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  7. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  8. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  11. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  12. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  13. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  14. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  15. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  16. 2022Rethinking and Refining the Distinct Metriccited by 27
  17. 2025Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Taskscited by 21
  18. 2024InstructPatentGPT: Training patent language models to follow instructions with human feedbackcited by 19
  19. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  20. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  21. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  22. 2024Towards Tailored Recovery of Lexical Diversity in Literary Machine Translationcited by 5
  23. 2024Instruction-tuned Large Language Models for Machine Translation in the Medical Domaincited by 4
  24. 2023Non-Autoregressive Document-Level Machine Translationcited by 1

Built on by

Not cited yet.