mlsanityData to 7 Oct 2026

QE-EBM: Using Quality Estimators as Energy Loss for Machine Translation

arXiv 2410.10228 ↗First posted 14 Oct 2024cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning has shown great promise in aligning language models with human preferences in a variety of text generation tasks, including machine translation. For translation tasks, rewards can easily be obtained from quality estimation (QE) models which can generate rewards for unlabeled data. Despite its usefulness, reinforcement learning cannot exploit the gradients with respect to the QE score. We propose QE-EBM, a method of employing quality estimators as trainable loss networks that can directly backpropagate to the NMT model. We examine our method on several low and high resource target languages with English as the source language. QE-EBM outperforms strong baselines such as REINFORCE and proximal policy optimization (PPO) as well as supervised fine-tuning for all target languages, especially low-resource target languages. Most notably, for English-to-Mongolian translation, our method achieves improvements of 2.5 BLEU, 7.1 COMET-KIWI, 5.3 COMET, and 6.4 XCOMET relative to the supervised baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 30 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

21 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model23 Jan
  2. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference15 Nov

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
7 more
  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  2. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  3. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  4. 2020Residual Energy-Based Models for Text Generationcited by 95
  5. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  6. 2022Reward Gaming in Conditional Text Generationcited by 29
  7. 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation
  2. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings
  3. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
  4. 2025SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translation
  5. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation
  6. 2026Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation
  7. 2026Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective
  8. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning