# MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors

arXiv: https://arxiv.org/abs/2607.00848 · HTML: https://mlsanity.com/paper/2607.00848/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 53 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 10 of 10)

- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [Improving LLM Abilities in Idiomatic Translation](https://mlsanity.com/paper/2407.03518/) (2024)
- [Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment](https://mlsanity.com/paper/2404.02490/) (2024)
- [MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language](https://mlsanity.com/paper/2406.13698/) (2024)
- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [Towards a resource for multilingual lexicons: an MT assisted and human-in-the-loop multilingual parallel corpus with multi-word expression annotation](https://mlsanity.com/paper/2011.03783/) (2020)
- [Non-Linear Scoring Model for Translation Quality Evaluation](https://mlsanity.com/paper/2511.13467/) (2025)
- [Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation](https://mlsanity.com/paper/2512.21787/) (2025)
- [Overview of MWE history, challenges, and horizons: standing at the 20th anniversary of the MWE workshop series via MWE-UD2024](https://mlsanity.com/paper/2412.18868/) (2024)

## Built on by (most cited first, 2 of 2)

- [Towards End-to-End Multilingual Metaphor Processing: Integrating Detection, Translation, and Evaluation](https://mlsanity.com/paper/2608.04260/) (2026)
- [Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus](https://mlsanity.com/paper/2609.06634/) (2026)

## Abstract (arXiv)

In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present challenges for machine translation (MT) and natural language understanding and processing (NLU, NLP), because it presents the features of semantic complexity, contextual dependency, and cultural embeddings that can lead to ambiguity issues for NLP models. To investigate how state-of-the-art NLP models perform on translating metaphors, we select three representative systems, i.e., GoogleMT, GPT5.4, and Hunyuan-7b as Neural MT (NMT) models and LLMs. We used two human-annotated metaphor corpora, including VUAMC and PSUCMC for English-to-Chinese and Chinese-to-English translation purposes. The original corpora we used are monolingual, where we carried out error annotation using the MetaHOPE framework, and also produced the human post-edited gold reference for bilingual use as a new resource. We believe the MetaHOPE evaluation framework for metaphor translation annotation, the parallel corpora resources, and the error analysis on SOTA automatic translation models can be useful and shed some light for the field of metaphor translation study. We share our resources publicly at github.com/Jiahui84/MetaHOPE