mlsanityData to 14 Oct 2026

IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages

arXiv 2212.10180 ↗First posted 20 Dec 2022cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

The rapid growth of machine translation (MT) systems has necessitated comprehensive studies to meta-evaluate evaluation metrics being used, which enables a better selection of metrics that best reflect MT quality. Unfortunately, most of the research focuses on high-resource languages, mainly English, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from English, and to date, there has not been a systematic study of evaluating MT systems from English into Indian languages. In this paper, we fill this gap by creating an MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, and use it to establish correlations between annotator scores and scores obtained using existing automatic metrics. Our results show that pre-trained metrics, such as COMET, have the highest correlations with annotator scores. Additionally, we find that the metrics do not adequately capture fluency-based errors in Indian languages, and there is a need to develop metrics focused on Indian languages. We hope that our dataset and analysis will help promote further research in this area.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

34 of 59 references.

References · 34

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 627
22 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  5. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  6. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  7. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  8. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  9. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  10. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  11. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  12. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  13. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  14. 2018Towards a Better Metric for Evaluating Question Generation Systemscited by 52
  15. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  16. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  17. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  18. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  19. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  20. 2018Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languagescited by 11
  21. 2018Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatiancited by 7
  22. 2020Utilizing Language Relatedness to improve Machine Translation: A Case Study on Languages of the Indian Subcontinentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 11; 2025: 7; 2026: 6.

27 papers.

Sort:

By area:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  2. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  3. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  4. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  5. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 16
  6. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  7. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  8. 2025SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?cited by 10
  9. 2024An Empirical Study of In-context Learning in LLMs for Machine Translationcited by 9
  10. 2024How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?cited by 6
  11. 2024Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Studycited by 5
  12. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 5
15 more
  1. 2024Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learningcited by 4
  2. 2024Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translationcited by 4
  3. 2024IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?cited by 2
  4. 2025Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPTcited by 2
  5. 2025Mark My Words: A Robust Multilingual Model for Punctuation in Text and Speech Transcriptscited by 2
  6. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  7. 2025L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languagescited by 1
  8. 2025Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpuscited by 1
  9. 2025PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generationcited by 1
  10. 2026PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translationcited by 1
  11. 2026IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languagescited by 1
  12. 2026Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025cited by 1
  13. 2025Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languagescited by 0
  14. 2026IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languagescited by 0
  15. 2026When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languagescited by 0