# LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

arXiv: https://arxiv.org/abs/2604.18490 · HTML: https://mlsanity.com/paper/2604.18490/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 64 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 17 of 17)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Benchmarking Machine Translation with Cultural Awareness](https://mlsanity.com/paper/2305.14328/) (2023)
- [MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning](https://mlsanity.com/paper/2504.10160/) (2025)
- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [DRT: Deep Reasoning Translation via Long Chain-of-Thought](https://mlsanity.com/paper/2412.17498/) (2024)
- [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2407.04910/) (2024)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [Casablanca: Data and Models for Multidialectal Arabic Speech Recognition](https://mlsanity.com/paper/2410.04527/) (2024)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://mlsanity.com/paper/2308.03051/) (2023)
- [Correct Me If You Can: Learning from Error Corrections and Markings](https://mlsanity.com/paper/2004.11222/) (2020)
- [Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost](https://mlsanity.com/paper/2510.20780/) (2025)

## Built on by (most cited first, 3 of 3)

- [AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation](https://mlsanity.com/paper/2609.22796/) (2026)
- [Enemray: Toward Capable Language Models for Hassaniya](https://mlsanity.com/paper/2609.14829/) (2026)
- [TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification](https://mlsanity.com/paper/2609.29633/) (2026)

## Abstract (arXiv)

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.