# Confident but Wrong: A Constrained Decoding Diagnostic for Low-Resource Automatic Post-Editing

arXiv: https://arxiv.org/abs/2609.29680 · HTML: https://mlsanity.com/paper/2609.29680/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'](https://mlsanity.com/paper/2410.12954/) (2024)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World](https://mlsanity.com/paper/2210.08523/) (2022)
- [Survey on Publicly Available Sinhala Natural Language Processing Tools and Research](https://mlsanity.com/paper/1906.02358/) (2019)
- [Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations](https://mlsanity.com/paper/2404.07851/) (2024)
- [Can Automatic Post-Editing Improve NMT?](https://mlsanity.com/paper/2009.14395/) (2020)
- [BERTifying Sinhala -- A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classification](https://mlsanity.com/paper/2208.07864/) (2022)
- [MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing](https://mlsanity.com/paper/1809.00188/) (2018)
- [Unbabel's Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing](https://mlsanity.com/paper/1905.13068/) (2019)
- [Learning to Copy for Automatic Post-Editing](https://mlsanity.com/paper/1911.03627/) (2019)
- [Giving the Old a Fresh Spin: Quality Estimation-Assisted Constrained Decoding for Automatic Post-Editing](https://mlsanity.com/paper/2501.17265/) (2025)
- [Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages](https://mlsanity.com/paper/2410.17973/) (2024)
- [EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation](https://mlsanity.com/paper/2609.29511/) (2026)

## Abstract (arXiv)

Automatic Post-Editing (APE) for low-resource languages (LRLs) often fails to improve Machine Translation (MT), and the score alone cannot say why: whether more training would help, or whether the training data is too inconsistent to learn from. We introduce a black-box, inference-time diagnostic that tells these two cases apart without retraining or annotation. It varies an edit-distance penalty λ that drives the model from free editing towards copying the MT, and reads two signals: (1) the shape of the Translation Edit Rate (TER)-vs-λ curve, U-shaped if edits from the model reduce error and monotonically decreasing if none does; and (2) the ordering of constraint variants that trust model confidence to increasing degrees, which shows whether confidence tracks edit quality. Across decoder-only and encoder-decoder models on English-Sinhala, the diagnostic exposes two failure modes consistent with a heterogeneous post-edit signal as the underlying cause: Binary Collapse, where the model copies the MT or makes off-target edits, and Confident Miscalibration, where the confidence signals we test do not separate useful edits from unnecessary ones. The pattern holds on English-Marathi and English-Tamil, with the failure modes tracking the post-edit distribution rather than MT quality or language family. Beyond diagnosis, the curve shape prescribes a concrete next step for practitioners; in the favorable case, a static constraint yields a free inference-time accuracy gain. We release the first English-Sinhala (~66k) and a new English-Tamil (~39k) APE datasets with all code.