mlsanityData to 14 Oct 2026

LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods

arXiv 2505.16129 ↗First posted 22 May 2025cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent studies have applied large language models (LLMs) to machine translation quality estimation (MTQE) by prompting models to assign numeric scores. Nonetheless, these direct scoring methods tend to show low segment-level correlation with human judgments. In this paper, we propose a generation-based evaluation paradigm that leverages decoder-only LLMs to produce high-quality references, followed by semantic similarity scoring using sentence embeddings. We conduct the most extensive evaluation to date in MTQE, covering 8 LLMs and 8 language pairs. Empirical results show that our method outperforms both intra-LLM direct scoring baselines and external non-LLM reference-free metrics from MTME. These findings demonstrate the strength of generation-based evaluation and support a shift toward hybrid approaches that combine fluent generation with accurate semantic assessment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 25 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

12 of 25 references.

References · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  10. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  11. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  12. 2024The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluatecited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation
  2. 2024What do Large Language Models Need for Machine Translation Evaluation?
  3. 2025Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends
  4. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
  5. 2023A Closer Look into Automatic Evaluation Using Large Language Models
  6. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  7. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
  8. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation