mlsanityData to 7 Oct 2026

IT5: Text-to-text Pretraining for Italian Language Understanding and Generation

arXiv 2203.03759 ↗First posted 7 Mar 2022cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

We introduce IT5, the first family of encoder-decoder transformer models pretrained specifically on Italian. We document and perform a thorough cleaning procedure for a large Italian corpus and use it to pretrain four IT5 model sizes. We then introduce the ItaGen benchmark, which includes a broad range of natural language understanding and generation tasks for Italian, and use it to evaluate the performance of IT5 models and multilingual baselines. We find monolingual IT5 models to provide the best scale-to-performance ratio across tested models, consistently outperforming their multilingual counterparts and setting a new state-of-the-art for Italian language generation.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

28 of 50 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
16 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  10. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  11. 2021XFORMAL: A Benchmark for Multilingual Formality Style Transfercited by 51
  12. 2020What the [MASK]? Making Sense of Language-Specific BERT Modelscited by 44
  13. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  14. 2020PTT5: Pretraining and validating the T5 model on Brazilian Portuguese datacited by 35
  15. 2020GePpeTto Carves Italian into a Language Modelcited by 16
  16. 2021On the interaction of automatic evaluation and task framing in headline style transfercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 2; 2025: 2; 2026: 4.

20 papers.

Sort:

By area:

  1. 2022ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generationcited by 37
  2. 2023Typhoon: Thai Large Language Modelscited by 28
  3. 2023LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Languagecited by 24
  4. 2023Fauno: The Italian Large Language Model that will leave you senza parole!cited by 13
  5. 2023Camoscio: an Italian Instruction-tuned LLaMAcited by 13
  6. 2023HeRo: RoBERTa and Longformer Hebrew Language Modelscited by 12
  7. 2023Legal Summarisation through LLMs: The PRODIGIT Projectcited by 10
  8. 2022Sequence to sequence pretraining for a less-resourced Slovenian languagecited by 9
  9. 2023Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?cited by 6
  10. 2023Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluationcited by 3
  11. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  12. 2022esCorpius: A Massive Spanish Crawling Corpuscited by 1
8 more
  1. 2024Annotation and Classification of Relevant Clauses in Terms-and-Conditions Contractscited by 1
  2. 2025PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generationcited by 1
  3. 2024Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)cited by 0
  4. 2025Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offscited by 0
  5. 2026TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entitiescited by 0
  6. 2026GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learningcited by 0
  7. 2026Language, Language Models, and What We're Talking Aboutcited by 0
  8. 2026Accelerating Sharded Data Parallelism at Scale with Federated Learningcited by 0