# ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation

arXiv: https://arxiv.org/abs/2606.05421 · HTML: https://mlsanity.com/paper/2606.05421/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Text Readability Assessment for Second Language Learners](https://mlsanity.com/paper/1906.07580/) (2019)
- [Controllable Text Simplification with Explicit Paraphrasing](https://mlsanity.com/paper/2010.11004/) (2020)
- [Linguistic Features for Readability Assessment](https://mlsanity.com/paper/2006.00377/) (2020)
- [Controlling Text Complexity in Neural Machine Translation](https://mlsanity.com/paper/1911.00835/) (2019)
- [Trends, Limitations and Open Challenges in Automatic Readability Assessment Research](https://mlsanity.com/paper/2105.00973/) (2021)
- [CEFR-Based Sentence Difficulty Annotation and Assessment](https://mlsanity.com/paper/2210.11766/) (2022)
- [It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information](https://mlsanity.com/paper/2005.02354/) (2020)
- [A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplification](https://mlsanity.com/paper/1810.05754/) (2018)
- [CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality](https://mlsanity.com/paper/2205.04022/) (2022)
- [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023)
- [An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages](https://mlsanity.com/paper/2109.00165/) (2021)
- [Analysing Zero-Shot Readability-Controlled Sentence Simplification](https://mlsanity.com/paper/2409.20246/) (2024)
- [Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation](https://mlsanity.com/paper/2005.11197/) (2020)
- [Estimating Machine Translation Difficulty](https://mlsanity.com/paper/2508.10175/) (2025)
- [UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment](https://mlsanity.com/paper/2506.01419/) (2025)
- [Machine Translation Models are Zero-Shot Detectors of Translation Direction](https://mlsanity.com/paper/2401.06769/) (2024)

## Abstract (arXiv)

When a text is translated, does the translation retain the complexity of the original? We introduce ComplexityMT, a new challenge for assessing how text complexity and machine translation interact with and influence each other, using the Common European Framework of Reference for Languages (CEFR) levels as the measure of text complexity. Across six languages, including Arabic, Dutch, English, French, Hindi, and Russian, we evaluate three open-weight models, one closed model, and a commercial machine translation system on two tasks: i) correlation of CEFR with translation difficulty, and ii) shifts in CEFR levels of the source texts. Our experiments show that higher CEFR levels make texts more difficult to translate and that machine translation lowers the target text's CEFR level for most languages at the document level. These findings provide new insights for researchers and practitioners working on multilingual pedagogical content generation and MT difficulty estimation.