# Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation

arXiv: https://arxiv.org/abs/2502.19941 · HTML: https://mlsanity.com/paper/2502.19941/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback](https://mlsanity.com/paper/2311.09336/) (2023)
- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://mlsanity.com/paper/1906.11455/) (2019)
- [Quality Estimation without Human-labeled Data](https://mlsanity.com/paper/2102.04020/) (2021)
- [BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment](https://mlsanity.com/paper/2406.12168/) (2024)
- [DirectQE: Direct Pretraining for Machine Translation Quality Estimation](https://mlsanity.com/paper/2105.07149/) (2021)
- [Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task](https://mlsanity.com/paper/2309.13230/) (2023)

## Built on by (most cited first, 4 of 4)

- [CMI-Mem: Toward Generalizable Long-Term Memory Management via CMI-Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.20553/) (2026)
- [Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation](https://mlsanity.com/paper/2608.03577/) (2026)
- [Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Error Span Detection](https://mlsanity.com/paper/2606.21559/) (2026)
- [Synthetic Data Characterization via Training Dynamics](https://mlsanity.com/paper/2609.39447/) (2026)

## Abstract (arXiv)

Quality Estimation (QE) models evaluate the quality of machine translations without reference translations, serving as the reward models for the translation task. Due to the data scarcity, synthetic data generation has emerged as a promising solution. However, synthetic QE data often suffers from distribution shift, which can manifest as discrepancies between pseudo and real translations, or in pseudo labels that do not align with human preferences. To tackle this issue, we introduce DCSQE, a novel framework for alleviating distribution shift in synthetic QE data. To reduce the difference between pseudo and real translations, we employ the constrained beam search algorithm and enhance translation diversity through the use of distinct generation models. DCSQE uses references, i.e., translation supervision signals, to guide both the generation and annotation processes, enhancing the quality of token-level labels. DCSQE further identifies the shortest phrase covering consecutive error tokens, mimicking human annotation behavior, to assign the final phrase-level labels. Specially, we underscore that the translation model can not annotate translations of itself accurately. Extensive experiments demonstrate that DCSQE outperforms SOTA baselines like CometKiwi in both supervised and unsupervised settings. Further analysis offers insights into synthetic data generation that could benefit reward models for other tasks. The code is available at https://github.com/NJUNLP/njuqe.