mlsanityData to 14 Oct 2026

TransEvalnia: Reasoning-based Evaluation and Ranking of Translations

arXiv 2507.12724 ↗First posted 17 Jul 2025cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present TransEvalnia, a prompting-based translation evaluation and ranking system that uses reasoning in performing its evaluations and ranking. This system presents fine-grained evaluations based on a subset of the Multidimensional Quality Metrics (https://themqm.org/), returns an assessment of which translation it deems the best, and provides numerical scores for the various dimensions and for the overall translation. We show that TransEvalnia performs as well as or better than the state-of-the-art MT-Ranker (Moosa et al. 2024) on our own English-Japanese data as well as several language pairs from various WMT shared tasks. Using Anthropic's Claude-3.5-Sonnet and Qwen-2.5-72B-Instruct as the evaluation LLMs, we show that the evaluations returned are deemed highly acceptable to human raters, and that the scores assigned to the translations by Sonnet, as well as other LLMs, correlate well with scores assigned by the human raters. We also note the sensitivity of our system -- as well as MT-Ranker -- to the order in which the translations are presented, and we propose methods to address this position bias. All data, including the system's evaluation and reasoning, human assessments, as well as code is released.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 35 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

13 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025New Trends for Modern Machine Translation with Large Reasoning Models13 Mar

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  4. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  5. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  6. 2024Critique-out-Loud Reward Modelscited by 106
  7. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  8. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  9. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  10. 2025Order Matters: Investigate the Position Bias in Multi-constraint Instruction Followingcited by 12
  11. 2025Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Modelcited by 8
  12. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling
  2. 2025LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods
  3. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation
  4. 2025TASER: Translation Assessment via Systematic Evaluation and Reasoning
  5. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
  6. 2026EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting
  7. 2025Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends
  8. 2025Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost