# AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

arXiv: https://arxiv.org/abs/2311.09828 · HTML: https://mlsanity.com/paper/2311.09828/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://mlsanity.com/paper/2010.04480/) (2020)
- [The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translation](https://mlsanity.com/paper/2103.08647/) (2021)
- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [yosm: A new yoruba sentiment corpus for movie reviews](https://mlsanity.com/paper/2204.09711/) (2022)
- ["Bilingual Expert" Can Find Translation Errors](https://mlsanity.com/paper/1807.09433/) (2018)
- [Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2109.07141/) (2021)
- [QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task](https://mlsanity.com/paper/2112.14890/) (2021)

## Built on by (most cited first, 24 of 24)

- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Tiny Aya: Bridging Scale and Multilingual Depth](https://mlsanity.com/paper/2603.11510/) (2026)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?](https://mlsanity.com/paper/2506.04557/) (2025)
- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [State of NLP in Kenya: A Survey](https://mlsanity.com/paper/2410.09948/) (2024)
- [Ibom NLP: A Step Toward Inclusive Natural Language Processing for Nigeria's Minority Languages](https://mlsanity.com/paper/2511.06531/) (2025)
- [Translation as a Scalable Proxy for Multilingual Evaluation](https://mlsanity.com/paper/2601.11778/) (2026)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis](https://mlsanity.com/paper/2412.05862/) (2024)
- [RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset](https://mlsanity.com/paper/2502.06180/) (2025)
- [LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama](https://mlsanity.com/paper/2503.11911/) (2025)
- [Exploring In-context Example Generation for Machine Translation](https://mlsanity.com/paper/2506.00507/) (2025)
- [AfriNLLB: Efficient Translation Models for African Languages](https://mlsanity.com/paper/2602.09373/) (2026)
- [Anthropogenic Regional Adaptation in Multimodal Vision-Language Model](https://mlsanity.com/paper/2604.11490/) (2026)
- [Bemba Speech Translation: Exploring a Low-Resource African Language](https://mlsanity.com/paper/2505.02518/) (2025)
- [OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages](https://mlsanity.com/paper/2508.16048/) (2025)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://mlsanity.com/paper/2509.20557/) (2025)
- [Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages](https://mlsanity.com/paper/2507.21568/) (2025)
- [The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP](https://mlsanity.com/paper/2510.05644/) (2025)
- [Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models](https://mlsanity.com/paper/2510.07203/) (2025)
- [TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation](https://mlsanity.com/paper/2608.18655/) (2026)
- [The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama](https://mlsanity.com/paper/2609.31727/) (2026)
- [Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning](https://mlsanity.com/paper/2609.40286/) (2026)

## Abstract (arXiv)

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).