# It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

arXiv: https://arxiv.org/abs/2005.02354 · HTML: https://mlsanity.com/paper/2005.02354/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 26 of 26)

- [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [Mutual Information Divergence: A Unified Metric for Multimodal Generative Models](https://mlsanity.com/paper/2205.13445/) (2022)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [A Large-Scale Study of Machine Translation in the Turkic Languages](https://mlsanity.com/paper/2109.04593/) (2021)
- [MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages](https://mlsanity.com/paper/2208.13078/) (2022)
- [A Principled Framework for Evaluating on Typologically Diverse Languages](https://mlsanity.com/paper/2407.05022/) (2024)
- [Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling](https://mlsanity.com/paper/2102.03551/) (2021)
- [DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages](https://mlsanity.com/paper/2205.12215/) (2022)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [On the Difficulty of Translating Free-Order Case-Marking Languages](https://mlsanity.com/paper/2107.06055/) (2021)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluation](https://mlsanity.com/paper/2402.19457/) (2024)
- [Estimating Machine Translation Difficulty](https://mlsanity.com/paper/2508.10175/) (2025)
- [Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents](https://mlsanity.com/paper/2305.13303/) (2023)
- [Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation](https://mlsanity.com/paper/2407.12832/) (2024)
- [Machine Translation Models are Zero-Shot Detectors of Translation Direction](https://mlsanity.com/paper/2401.06769/) (2024)
- [Specializing Multi-domain NMT via Penalizing Low Mutual Information](https://mlsanity.com/paper/2210.12910/) (2022)
- [Context-aware Neural Machine Translation for English-Japanese Business Scene Dialogues](https://mlsanity.com/paper/2311.11976/) (2023)
- [Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2602.03551/) (2026)
- [Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset](https://mlsanity.com/paper/2510.20508/) (2025)
- [Rashid: A Cipher-Based Framework for Exploring In-Context Language Learning](https://mlsanity.com/paper/2603.22497/) (2026)
- [ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation](https://mlsanity.com/paper/2606.05421/) (2026)
- [Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability](https://mlsanity.com/paper/2606.22269/) (2026)
- [Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition](https://mlsanity.com/paper/2609.12960/) (2026)

## Abstract (arXiv)

The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.