# IT5: Text-to-text Pretraining for Italian Language Understanding and Generation

arXiv: https://arxiv.org/abs/2203.03759 · HTML: https://mlsanity.com/paper/2203.03759/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering](https://mlsanity.com/paper/2010.12643/) (2020)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [GePpeTto Carves Italian into a Language Model](https://mlsanity.com/paper/2004.14253/) (2020)
- [On the interaction of automatic evaluation and task framing in headline style transfer](https://mlsanity.com/paper/2101.01634/) (2021)

## Built on by (most cited first, 20 of 20)

- [ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation](https://mlsanity.com/paper/2205.06457/) (2022)
- [Typhoon: Thai Large Language Models](https://mlsanity.com/paper/2312.13951/) (2023)
- [LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language](https://mlsanity.com/paper/2312.09993/) (2023)
- [Fauno: The Italian Large Language Model that will leave you senza parole!](https://mlsanity.com/paper/2306.14457/) (2023)
- [Camoscio: an Italian Instruction-tuned LLaMA](https://mlsanity.com/paper/2307.16456/) (2023)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://mlsanity.com/paper/2304.11077/) (2023)
- [Legal Summarisation through LLMs: The PRODIGIT Project](https://mlsanity.com/paper/2308.04416/) (2023)
- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)
- [Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?](https://mlsanity.com/paper/2305.15908/) (2023)
- [Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation](https://mlsanity.com/paper/2311.15698/) (2023)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [esCorpius: A Massive Spanish Crawling Corpus](https://mlsanity.com/paper/2206.15147/) (2022)
- [Annotation and Classification of Relevant Clauses in Terms-and-Conditions Contracts](https://mlsanity.com/paper/2402.14457/) (2024)
- [PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation](https://mlsanity.com/paper/2512.13552/) (2025)
- [Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)](https://mlsanity.com/paper/2402.17608/) (2024)
- [Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offs](https://mlsanity.com/paper/2509.13480/) (2025)
- [TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities](https://mlsanity.com/paper/2603.27768/) (2026)
- [GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learning](https://mlsanity.com/paper/2606.01386/) (2026)
- [Language, Language Models, and What We're Talking About](https://mlsanity.com/paper/2609.03577/) (2026)
- [Accelerating Sharded Data Parallelism at Scale with Federated Learning](https://mlsanity.com/paper/2609.20359/) (2026)

## Abstract (arXiv)

We introduce IT5, the first family of encoder-decoder transformer models pretrained specifically on Italian. We document and perform a thorough cleaning procedure for a large Italian corpus and use it to pretrain four IT5 model sizes. We then introduce the ItaGen benchmark, which includes a broad range of natural language understanding and generation tasks for Italian, and use it to evaluate the performance of IT5 models and multilingual baselines. We find monolingual IT5 models to provide the best scale-to-performance ratio across tested models, consistently outperforming their multilingual counterparts and setting a new state-of-the-art for Italian language generation.