mlsanityData to 7 Oct 2026

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

arXiv 2206.11249 ↗First posted 22 Jun 2022cs.CL cs.AI cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 61 of its 85 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

61 of 85 references.

References · 61

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  9. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  10. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  11. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  12. 2019Abductive Commonsense Reasoningcited by 300
38 more
  1. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  2. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  3. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  4. 2017Challenges in Data-to-Document Generationcited by 226
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  6. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  7. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  8. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  9. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  10. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  11. 2021The Benchmark Lotterycited by 133
  12. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  13. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  14. 2020Text-to-Text Pre-Training for Data-to-Text Taskscited by 104
  15. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  16. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  17. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 79
  18. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  19. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  20. 2020CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Datasetcited by 66
  21. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  22. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  23. 2021Paragraph-level Simplification of Medical Textscited by 52
  24. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  25. 2021D2S: Document-to-Slide Generation Via Query-Based Text Summarizationcited by 48
  26. 2019Data-to-text Generation with Entity Modelingcited by 45
  27. 2020RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modelingcited by 40
  28. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 39
  29. 2022Models and Datasets for Cross-Lingual Summarisationcited by 36
  30. 2019Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialoguecited by 35
  31. 2021Data-to-text Generation with Macro Planningcited by 34
  32. 2021A Systematic Review of Reproducibility Research in Natural Language Processingcited by 34
  33. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  34. 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
  35. 2021"How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversationscited by 30
  36. 2018Open Subtitles Paraphrase Corpus for Six Languagescited by 24
  37. 2021Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cardscited by 24
  38. 2019Generating Summaries with Topic Templates and Structured Convolutional Decoderscited by 20

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 10; 2024: 9; 2025: 2; 2026: 1.

36 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  3. 2022RealTime QA: What's the Answer Right Now?cited by 239
  4. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  5. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  6. 2022Evaluating Human-Language Model Interactioncited by 90
  7. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  8. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  9. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  10. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  11. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  12. 2023Evaluating General-Purpose AI with Psychometricscited by 23
24 more
  1. 2022CiteBench: A benchmark for Scientific Citation Text Generationcited by 22
  2. 2023Dolphin: A Challenging and Diverse Benchmark for Arabic NLGcited by 22
  3. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  4. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  5. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  6. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  7. 2022BanglaParaphrase: A High-Quality Bangla Paraphrase Datasetcited by 15
  8. 2024Cheetah: Natural Language Generation for 517 African Languagescited by 15
  9. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  10. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  11. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12
  12. 2022Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corporacited by 11
  13. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  14. 2023Cross-Lingual Supervision improves Large Language Models Pre-trainingcited by 8
  15. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 8
  16. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  17. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 4
  18. 2024InspectorRAGet: An Introspection Platform for RAG Evaluationcited by 3
  19. 2026Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checkingcited by 3
  20. 2022CLSE: Corpus of Linguistically Significant Entitiescited by 2
  21. 2022A Major Obstacle for NLP Research: Let's Talk about Time Allocation!cited by 1
  22. 2022Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Modelscited by 1
  23. 2025Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?cited by 1
  24. 2022Evaluation for Changecited by 0