mlsanityData to 7 Oct 2026

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

arXiv 2102.01672 ↗First posted 2 Feb 2021cs.CL cs.AI cs.LGcited by 239

Not on the map. Why?

Abstract (arXiv)

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.

Key facts. Cited by 239 papers in the map (data through 7 Oct 2026). Builds on 89 of its 171 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

89 of 171 references.

References · 89

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
38 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  4. 2018Model Cards for Model Reportingcited by 1,068
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  6. 2015Teaching Machines to Read and Comprehendcited by 1,039
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  10. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  11. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  13. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  14. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  15. 2018CoQA: A Conversational Question Answering Challengecited by 674
  16. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  17. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  18. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  19. 2019ELI5: Long Form Question Answeringcited by 567
  20. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  21. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  22. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  23. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  24. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  25. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  26. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  27. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  28. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  29. 2020Evaluation of Text Generation: A Surveycited by 275
  30. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  31. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  32. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  33. 2017Challenges in Data-to-Document Generationcited by 226
  34. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  35. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  36. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  37. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  38. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 55; 2022: 82; 2023: 38; 2024: 30; 2025: 16; 2026: 18.

239 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2022PAL: Program-aided Language Modelscited by 635
  8. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  12. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
38 more
  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2023Scaling Data-Constrained Language Modelscited by 391
  3. 2022UL2: Unifying Language Learning Paradigmscited by 292
  4. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  5. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  6. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  7. 2022RealTime QA: What's the Answer Right Now?cited by 239
  8. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  9. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  10. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  12. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  13. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  14. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  15. 2021The Benchmark Lotterycited by 133
  16. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  17. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  18. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  19. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 122
  20. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  21. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  22. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  23. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  24. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  25. 2022Evaluating Human-Language Model Interactioncited by 90
  26. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  27. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  28. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  29. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  30. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  31. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  32. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  33. 2023Efficient Benchmarking of Language Modelscited by 71
  34. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  35. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  36. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  37. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 63
  38. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62

50 most cited of 239 All 239 (JSON)