mlsanityData to 7 Oct 2026

MQM-Chat: Multidimensional Quality Metrics for Chat Translation

arXiv 2408.16390 ↗First posted 29 Aug 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat). Through the experiments of five models using MQM-Chat, we observed that all models generated certain fundamental errors, while each of them has different shortcomings, such as omission, overly correcting ambiguous source content, and buzzword issues, resulting in the loss of stylized information. Our findings underscore the effectiveness of MQM-Chat in evaluating chat translation, emphasizing the importance of stylized content and dialogue consistency for future studies.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

33 of 65 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2017Convolutional Sequence to Sequence Learningcited by 550
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
21 more
  1. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  2. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  3. 2017Neural Machine Translation with Extended Contextcited by 95
  4. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  5. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  6. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  7. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  8. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58
  9. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  10. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  11. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  12. 2017Document Context Neural Machine Translation with Memory Networkscited by 41
  13. 2021NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven Conversationcited by 35
  14. 2021Empirical Analysis of Training Strategies of Transformer-based Japanese Chit-chat Systemscited by 19
  15. 2021Modeling Bilingual Conversational Characteristics for Neural Chat Translationcited by 18
  16. 2023An Empirical Study of Translation Hypothesis Ensembling with Large Language Modelscited by 16
  17. 2018Contextual Neural Model for Translating Bilingual Multi-Speaker Conversationscited by 15
  18. 2021Towards Making the Most of Dialogue Characteristics for Neural Chat Translationcited by 12
  19. 2022Scheduled Multi-task Learning for Neural Chat Translationcited by 7
  20. 2023Chat Translation Error Detection for Assisting Cross-lingual Communicationscited by 3
  21. 2024An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communicationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3
  2. 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Modelscited by 2
  3. 2026An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile Devicescited by 0