mlsanityData to 7 Oct 2026

Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation

arXiv 2406.11580 ↗First posted 17 Jun 2024cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

High-quality Machine Translation (MT) evaluation relies heavily on human judgments. Comprehensive error classification methods, such as Multidimensional Quality Metrics (MQM), are expensive as they are time-consuming and can only be done by experts, whose availability may be limited especially for low-resource languages. On the other hand, just assigning overall scores, like Direct Assessment (DA), is simpler and faster and can be done by translators of any level, but is less reliable. In this paper, we introduce Error Span Annotation (ESA), a human evaluation protocol which combines the continuous rating of DA with the high-level error severity span marking of MQM. We validate ESA by comparing it to MQM and DA for 12 MT systems and one human reference translation (English to German) from WMT23. The results show that ESA offers faster and cheaper annotations than MQM at the same quality level, without the requirement of expensive MQM experts.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 8 of its 29 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

8 of 29 references.

References · 8

  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  3. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  4. 2018Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MTcited by 21
  5. 2024Finding Replicable Human Evaluations via Stable Ranking Probabilitycited by 13
  6. 2020Correct Me If You Can: Learning from Error Corrections and Markingscited by 11
  7. 2018Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatiancited by 7
  8. 2022Searching for a higher power in the human evaluation of MTcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 32; 2026: 24.

62 papers.

Sort:

By area:

  1. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  2. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  3. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  4. 2024Preliminary WMT24 Ranking of General MT Systems and LLMscited by 20
  5. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  6. 2025SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translationcited by 13
  7. 2025AFRIDOC-MT: Document-level MT Corpus for African Languagescited by 11
  8. 2024Pitfalls and Outlooks in Using COMETcited by 10
  9. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  10. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  11. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  12. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
38 more
  1. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9
  2. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7
  3. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  4. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  5. 2025Estimating Machine Translation Difficultycited by 6
  6. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
  7. 2024Localizing and Mitigating Errors in Long-form Question Answeringcited by 4
  8. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  9. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  10. 2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
  11. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  12. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  13. 2026LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translationcited by 3
  14. 2025Early-Exit and Instant Confidence Translation Quality Estimationcited by 2
  15. 2025How Important is `Perfect' English for Machine Translation Prompts?cited by 2
  16. 2025COMET-poly: Machine Translation Metric Grounded in Other Candidatescited by 2
  17. 2025Long-context Reference-based MT Quality Estimationcited by 2
  18. 2025MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translationcited by 2
  19. 2025Low-Resource, High-Impact: Building Corpora for Inclusive Language Technologiescited by 2
  20. 2025Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translationcited by 2
  21. 2026Strategies for Span Labeling with Large Language Modelscited by 2
  22. 2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2
  23. 2025TranslationCorrect: A Unified Framework for Machine Translation Post-Editing with Predictive Error Assistancecited by 1
  24. 2025Preliminary Ranking of WMT25 General Machine Translation Systemscited by 1
  25. 2025SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languagescited by 1
  26. 2025Generating Difficult-to-Translate Textscited by 1
  27. 2026PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translationcited by 1
  28. 2026Quantifying the Impact of Translation Errors on Multilingual LLM Evaluationcited by 1
  29. 2026Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intentcited by 1
  30. 2026Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translationcited by 1
  31. 2026Contrastive ESA: Human Evaluation of Multiple Translations at Oncecited by 1
  32. 2026Last Translation Benchmarkcited by 1
  33. 2025Real-World Summarization: When Evaluation Reaches Its Limitscited by 0
  34. 2025Audio-Based Crowd-Sourced Evaluation of Machine Translation Qualitycited by 0
  35. 2025Structured Document Translation via Format Reinforcement Learningcited by 0
  36. 2026Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translationcited by 0
  37. 2026Large Language Models as Annotators for Machine Translation Quality Estimationcited by 0
  38. 2026Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translationcited by 0

50 most cited of 62 All 62 (JSON)