mlsanityData to 14 Oct 2026

Beyond BLEU: Training Neural Machine Translation with Semantic Similarity

arXiv 1909.06694 ↗First posted 14 Sep 2019cs.CLcited by 84

Not on the map. Why?

Abstract (arXiv)

While most neural machine translation (NMT) systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can substantially improve final translation accuracy. However, training with BLEU has some limitations: it doesn't assign partial credit, it has a limited range of output values, and it can penalize semantically correct hypotheses if they differ lexically from the reference. In this paper, we introduce an alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity. We evaluate on four disparate languages translated to English, and find that training with our proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and also that the optimization procedure converges faster. Analysis suggests that this is because the proposed metric is more conducive to optimization, assigning partial credit and providing more diversity in scores than BLEU.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 17; 2023: 25; 2024: 17; 2025: 10; 2026: 3.

84 papers.

Sort:

By area:

  1. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
  2. 2022Confident Adaptive Language Modelingcited by 310
  3. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  4. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  5. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  6. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  7. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  8. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  9. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  10. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  11. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  12. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
38 more
  1. 2021Text Detoxification using Large Pre-trained Neural Modelscited by 52
  2. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  3. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  4. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  5. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  6. 2023You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Contentcited by 39
  7. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 33
  8. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  9. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  10. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  11. 2021Paraphrastic Representations at Scalecited by 21
  12. 2022RELIC: Retrieving Evidence for Literary Claimscited by 18
  13. 2021Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generationcited by 17
  14. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  15. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16
  16. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  17. 2021Don't Go Far Off: An Empirical Study on Neural Poetry Translationcited by 15
  18. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  19. 2023Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxificationcited by 15
  20. 2023Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based Learningcited by 14
  21. 2023A Call for Standardization and Validation of Text Style Transfer Evaluationcited by 14
  22. 2023STEER: Unified Style Transfer with Expert Reinforcementcited by 14
  23. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solutioncited by 13
  24. 2021Reward Optimization for Neural Machine Translation with Learned Metricscited by 12
  25. 2022Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledgecited by 12
  26. 2025Vision Language Models in Medicinecited by 12
  27. 2024Authorship Style Transfer with Policy Optimizationcited by 11
  28. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  29. 2023Improving Factuality of Abstractive Summarization via Contrastive Reward Learningcited by 9
  30. 2024Natural Language Processing RELIES on Linguisticscited by 9
  31. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  32. 2022So Different Yet So Alike! Constrained Unsupervised Text Style Transfercited by 7
  33. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  34. 2022T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representationcited by 6
  35. 2023DiffuDetox: A Mixed Diffusion Model for Text Detoxificationcited by 6
  36. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  37. 2023CMD: a framework for Context-aware Model self-Detoxificationcited by 6
  38. 2025Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Taskscited by 6

50 most cited of 84 All 84 (JSON)