# GLGE: A New General Language Generation Evaluation Benchmark

arXiv: https://arxiv.org/abs/2011.11928 · HTML: https://mlsanity.com/paper/2011.11928/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 30 of 47)

- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla](https://mlsanity.com/paper/2205.11081/) (2022)
- [LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation](https://mlsanity.com/paper/2108.12960/) (2021)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Learning to Transfer Prompts for Text Generation](https://mlsanity.com/paper/2205.01543/) (2022)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.04044/) (2023)
- [Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers](https://mlsanity.com/paper/2108.00308/) (2021)
- [GEM: A General Evaluation Benchmark for Multimodal Tasks](https://mlsanity.com/paper/2106.09889/) (2021)
- [CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2112.13610/) (2021)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning](https://mlsanity.com/paper/2407.05399/) (2024)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation](https://mlsanity.com/paper/2210.13304/) (2022)
- [NoCoLA: The Norwegian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2306.07790/) (2023)
- [MuLD: The Multitask Long Document Benchmark](https://mlsanity.com/paper/2202.07362/) (2022)
- [Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://mlsanity.com/paper/2304.11791/) (2023)
- [Language Generation with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2405.18906/) (2024)
- [LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://mlsanity.com/paper/2410.09527/) (2024)
- [Improving Non-autoregressive Generation with Mixup Training](https://mlsanity.com/paper/2110.11115/) (2021)
- [EL-Attention: Memory Efficient Lossless Attention for Generation](https://mlsanity.com/paper/2105.04779/) (2021)
- [STORYWARS: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation](https://mlsanity.com/paper/2305.08152/) (2023)
- [Asking Questions Like Educational Experts: Automatically Generating Question-Answer Pairs on Real-World Examination Data](https://mlsanity.com/paper/2109.05179/) (2021)
- [Joint Generator-Ranker Learning for Natural Language Generation](https://mlsanity.com/paper/2206.13974/) (2022)

## Abstract (arXiv)

Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet (The source code and dataset are publicly available at https://github.com/microsoft/glge).