# MTG: A Benchmark Suite for Multilingual Text Generation

arXiv: https://arxiv.org/abs/2108.07140 · HTML: https://mlsanity.com/paper/2108.07140/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [NCLS: Neural Cross-Lingual Summarization](https://mlsanity.com/paper/1909.00156/) (2019)
- [Story Ending Generation with Incremental Encoding and Commonsense Knowledge](https://mlsanity.com/paper/1808.10113/) (2018)
- [Variational Template Machine for Data-to-Text Generation](https://mlsanity.com/paper/2002.01127/) (2020)
- [Unsupervised Pivot Translation for Distant Languages](https://mlsanity.com/paper/1906.02461/) (2019)

## Built on by (most cited first, 22 of 22)

- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [LongForm: Effective Instruction Tuning with Reverse Instructions](https://mlsanity.com/paper/2304.08460/) (2023)
- [Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.17050/) (2023)
- [Text Embedding Inversion Security for Multilingual Language Models](https://mlsanity.com/paper/2401.12192/) (2024)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [Large Language Models Are Cross-Lingual Knowledge-Free Reasoners](https://mlsanity.com/paper/2406.16655/) (2024)
- [Model Inversion Attacks: A Survey of Approaches and Countermeasures](https://mlsanity.com/paper/2411.10023/) (2024)
- [A Practical Examination of AI-Generated Text Detectors for Large Language Models](https://mlsanity.com/paper/2412.05139/) (2024)
- [xPQA: Cross-Lingual Product Question Answering across 12 Languages](https://mlsanity.com/paper/2305.09249/) (2023)
- [M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2502.11824/) (2025)
- [LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://mlsanity.com/paper/2410.09527/) (2024)
- [Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors](https://mlsanity.com/paper/2503.15128/) (2025)
- [Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis](https://mlsanity.com/paper/2410.13237/) (2024)
- [Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators](https://mlsanity.com/paper/2503.04360/) (2025)
- [SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languages](https://mlsanity.com/paper/2412.01186/) (2024)
- [SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning](https://mlsanity.com/paper/2407.13297/) (2024)
- [Extrapolating Multilingual Understanding Models as Multilingual Generators](https://mlsanity.com/paper/2305.13140/) (2023)
- [Multi-teacher Distillation for Multilingual Spelling Correction](https://mlsanity.com/paper/2311.11518/) (2023)
- [Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?](https://mlsanity.com/paper/2507.20419/) (2025)
- [The Imperfective Paradox in Large Language Models](https://mlsanity.com/paper/2601.09373/) (2026)
- [RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism](https://mlsanity.com/paper/2605.25565/) (2026)

## Abstract (arXiv)

We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It includes four generation tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). The multiway setup enables testing knowledge transfer capabilities for a model across languages and tasks. Using MTG, we train and analyze several popular multilingual generation models from different aspects. Our benchmark suite fosters model performance enhancement with more human-annotated parallel data. It provides comprehensive evaluations with diverse generation scenarios. Code and data are available at urlhttps://github.com/zide05/MTG.