# SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages

arXiv: https://arxiv.org/abs/2509.20557 · HTML: https://mlsanity.com/paper/2509.20557/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 94 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Translationese as a Language in "Multilingual" NMT](https://mlsanity.com/paper/1911.03823/) (2019)
- [Speech-to-Speech Translation For A Real-world Unwritten Language](https://mlsanity.com/paper/2211.06474/) (2022)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction](https://mlsanity.com/paper/2010.03260/) (2020)
- [Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?](https://mlsanity.com/paper/2203.08850/) (2022)
- [AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages](https://mlsanity.com/paper/2311.09828/) (2023)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary](https://mlsanity.com/paper/1811.01137/) (2018)
- [Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation](https://mlsanity.com/paper/2408.12780/) (2024)
- [SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?](https://mlsanity.com/paper/2506.04557/) (2025)
- [Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien](https://mlsanity.com/paper/2301.08937/) (2023)
- [Regressive Ensemble for Machine Translation Quality Evaluation](https://mlsanity.com/paper/2109.07242/) (2021)
- [How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?](https://mlsanity.com/paper/2406.03893/) (2024)
- [Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning](https://mlsanity.com/paper/2506.04408/) (2025)
- [Punctuation Restoration Improves Structure Understanding Without Supervision](https://mlsanity.com/paper/2402.08382/) (2024)
- [Learning from others' mistakes: Finetuning machine translation models with span-level error annotations](https://mlsanity.com/paper/2410.16509/) (2024)

## Built on by (most cited first, 1 of 1)

- [textscCantoNLU: A benchmark for Cantonese natural language understanding](https://mlsanity.com/paper/2510.20670/) (2025)

## Abstract (arXiv)

Despite major advances in machine translation (MT) in recent years, progress remains limited for many low-resource languages that lack large-scale training data and linguistic resources. In this paper, we introduce dsname, a novel fine-grained dataset that builds on existing parallel corpora to provide error span, error type, and error severity annotations in machine-translated examples from English to Mandarin, Cantonese, and Wu Chinese, along with a Mandarin-Hokkien component derived from a non-parallel source. Our dataset serves as a resource for the MT community to fine-tune models with error detection capabilities, supporting research on translation quality estimation, error-aware generation, and low-resource language evaluation. We also establish baseline results using language models to benchmark translation error detection performance. Specifically, we evaluate multiple open source and closed source LLMs using span-level and correlation-based MQM metrics, revealing their limited precision, underscoring the need for our dataset. Finally, we report our rigorous annotation process by native speakers, with analyses on pilot studies, iterative feedback, insights, and patterns in error type and severity.