# mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

arXiv: https://arxiv.org/abs/2305.11129 · HTML: https://mlsanity.com/paper/2305.11129/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 15 of 15)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)

## Built on by (most cited first, 10 of 10)

- [A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges](https://mlsanity.com/paper/2412.04782/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [An Open Multilingual System for Scoring Readability of Wikipedia](https://mlsanity.com/paper/2406.01835/) (2024)
- [When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages](https://mlsanity.com/paper/2501.04473/) (2025)
- [HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew](https://mlsanity.com/paper/2406.03897/) (2024)
- [German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset](https://mlsanity.com/paper/2403.03750/) (2024)
- [Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation](https://mlsanity.com/paper/2407.18789/) (2024)
- [AIstorian lets AI be a historian: A KG-powered multi-agent system for accurate biography generation](https://mlsanity.com/paper/2503.11346/) (2025)
- [The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages](https://mlsanity.com/paper/2507.18762/) (2025)

## Abstract (arXiv)

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.