# Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost

arXiv: https://arxiv.org/abs/2510.20780 · HTML: https://mlsanity.com/paper/2510.20780/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 1 of its 48 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 1 of 1)

- [Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2411.13775/) (2024)

## Built on by (most cited first, 4 of 4)

- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://mlsanity.com/paper/2601.07338/) (2026)
- [LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation](https://mlsanity.com/paper/2604.18490/) (2026)
- [Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged](https://mlsanity.com/paper/2610.10918/) (2026)
- [Retrieve What Helps, Only When It Helps: Agentic Reinforcement Learning for Selective Retrieval-Augmented Machine Translation](https://mlsanity.com/paper/2610.19528/) (2026)

## Abstract (arXiv)

Recent advancements in large reasoning models (LRMs) have introduced an intermediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of LRM-as-a-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~35x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g., R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These findings highlight the potential of efficiently calibrated LRMs to advance fine-grained automatic MT evaluation.