# Unbabel's Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing

arXiv: https://arxiv.org/abs/1905.13068 · HTML: https://mlsanity.com/paper/1905.13068/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 8 of 8)

- [Netmarble AI Center's WMT21 Automatic Post-Editing Shared Task Submission](https://mlsanity.com/paper/2109.06515/) (2021)
- [BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation](https://mlsanity.com/paper/2111.06787/) (2021)
- [Giving the Old a Fresh Spin: Quality Estimation-Assisted Constrained Decoding for Automatic Post-Editing](https://mlsanity.com/paper/2501.17265/) (2025)
- [Bilingual Synchronization: Restoring Translational Relationships with Editing Operations](https://mlsanity.com/paper/2210.13163/) (2022)
- [Automatic Correction of Human Translations](https://mlsanity.com/paper/2206.08593/) (2022)
- [Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages](https://mlsanity.com/paper/2410.17973/) (2024)
- [APE-then-QE: Correcting then Filtering Pseudo Parallel Corpora for MT Training Data Creation](https://mlsanity.com/paper/2312.11312/) (2023)
- [Confident but Wrong: A Constrained Decoding Diagnostic for Low-Resource Automatic Post-Editing](https://mlsanity.com/paper/2609.29680/) (2026)

## Abstract (arXiv)

This paper describes Unbabel's submission to the WMT2019 APE Shared Task for the English-German language pair. Following the recent rise of large, powerful, pre-trained models, we adapt the BERT pretrained model to perform Automatic Post-Editing in an encoder-decoder framework. Analogously to dual-encoder architectures we develop a BERT-based encoder-decoder (BED) model in which a single pretrained BERT encoder receives both the source src and machine translation tgt strings. Furthermore, we explore a conservativeness factor to constrain the APE system to perform fewer edits. As the official results show, when trained on a weighted combination of in-domain and artificial training data, our BED system with the conservativeness penalty improves significantly the translations of a strong Neural Machine Translation system by -0.78 and +1.23 in terms of TER and BLEU, respectively. Finally, our submission achieves a new state-of-the-art, ex-aequo, in English-German APE of NMT.