# Augmenting Text to Increase Translation Difficulty

arXiv: https://arxiv.org/abs/2608.15932 · HTML: https://mlsanity.com/paper/2608.15932/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 16 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!](https://mlsanity.com/paper/1809.08731/) (2018)
- [Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs](https://mlsanity.com/paper/2506.17080/) (2025)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation](https://mlsanity.com/paper/2602.16763/) (2026)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://mlsanity.com/paper/2408.13831/) (2024)
- [Generating Natural Language Adversarial Examples on a Large Scale with Generative Models](https://mlsanity.com/paper/2003.10388/) (2020)
- [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://mlsanity.com/paper/2504.01001/) (2025)
- [Pearmut: Human Evaluation of Translation Made Trivial](https://mlsanity.com/paper/2601.02933/) (2026)
- [Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Model](https://mlsanity.com/paper/1909.07873/) (2019)
- [Fine-grained evaluation of German-English Machine Translation based on a Test Suite](https://mlsanity.com/paper/1910.07460/) (2019)
- [BeamAttack: Generating High-quality Textual Adversarial Examples through Beam Search and Mixed Semantic Spaces](https://mlsanity.com/paper/2303.07199/) (2023)
- [A Relaxed Optimization Approach for Adversarial Attacks against Neural Machine Translation Models](https://mlsanity.com/paper/2306.08492/) (2023)

## Built on by (most cited first, 1 of 1)

- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)

## Abstract (arXiv)

As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality. We propose augmenting existing benchmarks to increase translation difficulty by combining adversarial optimization with a differentiable translation difficulty estimator. Our Adversarial Translation Optimization (ATO) uses gradients from a combined difficulty and fluency objective to iteratively replace tokens. Because each step branches over candidate substitutions at every position, optimization becomes a tree search problem, which we address with Beam Search. ATO offers a gradient-based alternative to LLM-based dataset creation without LLM prompting, expensive human curation, or task-specific model training. Our ATO-modified benchmark lowers average translation quality (xCOMET) from 0.93 to 0.82, compared to 0.88 for paraphrasing and 0.86 for a zero-shot baseline. Human evaluation shows the modified texts are somewhat less natural than the baselines but remain reasonably grammatical and plausible while being substantially harder to translate. We release two datasets of 350 English texts each, generated by our methods, as well as the code.