mlsanityData to 14 Oct 2026

A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation

arXiv 2106.06292 ↗First posted 11 Jun 2021cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Recent advances in AI and ML applications have benefited from rapid progress in NLP research. Leaderboards have emerged as a popular mechanism to track and accelerate progress in NLP through competitive model development. While this has increased interest and participation, the over-reliance on single, and accuracy-based metrics have shifted focus from other important metrics that might be equally pertinent to consider in real-world contexts. In this paper, we offer a preliminary discussion of the risks associated with focusing exclusively on accuracy metrics and draw on recent discussions to highlight prescriptive suggestions on how to develop more practical and effective leaderboards that can better reflect the real-world utility of models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

29 of 48 references.

References · 29

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2018Model Cards for Model Reportingcited by 1,080
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
17 more
  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  3. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  4. 2017Six Challenges for Neural Machine Translationcited by 335
  5. 2019Evaluating Gender Bias in Machine Translationcited by 249
  6. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  7. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  8. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  9. 2016Can neural machine translation do simultaneous translation?cited by 86
  10. 2019Domain Robustness in Neural Machine Translationcited by 65
  11. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  12. 2018Towards Robust Neural Machine Translationcited by 35
  13. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  14. 2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28
  15. 2021How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluationcited by 24
  16. 2019Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communitiescited by 19
  17. 2018An Operation Sequence Model for Explainable Neural Machine Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Position Paper on the Automatic Generation of Machine Learning Leaderboardscited by 3