# PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data

arXiv: https://arxiv.org/abs/2008.09144 · HTML: https://mlsanity.com/paper/2008.09144/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2020.

## Built on by (most cited first, 30 of 35)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Cabrita: closing the gap for foreign languages](https://mlsanity.com/paper/2308.11878/) (2023)
- [IT5: Text-to-text Pretraining for Italian Language Understanding and Generation](https://mlsanity.com/paper/2203.03759/) (2022)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)
- [Gl'orIA -- A Generative and Open Large Language Model for Portuguese](https://mlsanity.com/paper/2402.12969/) (2024)
- [Evaluation of Transfer Learning for Polish with a Text-to-Text Model](https://mlsanity.com/paper/2205.08808/) (2022)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task](https://mlsanity.com/paper/2205.15172/) (2022)
- [ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language](https://mlsanity.com/paper/2406.10806/) (2024)
- [A cost-benefit analysis of cross-lingual transfer methods](https://mlsanity.com/paper/2105.06813/) (2021)
- [Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy](https://mlsanity.com/paper/2412.09415/) (2024)
- [DEEPAG'E: Answering Questions in Portuguese about the Brazilian Environment](https://mlsanity.com/paper/2110.10015/) (2021)
- [PLSUM: Generating PT-BR Wikipedia by Summarizing Multiple Websites](https://mlsanity.com/paper/2112.01591/) (2021)
- [Sequence-to-Sequence Models for Extracting Information from Registration and Legal Documents](https://mlsanity.com/paper/2201.05658/) (2022)
- [Integrating question answering and text-to-SQL in Portuguese](https://mlsanity.com/paper/2202.04048/) (2022)
- [Tweets2Stance: Users stance detection exploiting Zero-Shot Learning Algorithms on Tweets](https://mlsanity.com/paper/2204.10710/) (2022)
- [Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes](https://mlsanity.com/paper/2211.05015/) (2022)
- [Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers](https://mlsanity.com/paper/2212.00587/) (2022)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [DeBERTinha: A Multistep Approach to Adapt DebertaV3 XSmall for Brazilian Portuguese Natural Language Processing Task](https://mlsanity.com/paper/2309.16844/) (2023)
- [Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts dataset](https://mlsanity.com/paper/2403.12212/) (2024)
- [BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives](https://mlsanity.com/paper/2201.03382/) (2022)
- [MonoByte: A Pool of Monolingual Byte-level Language Models](https://mlsanity.com/paper/2209.11035/) (2022)
- [Local Structure Matters Most in Most Languages](https://mlsanity.com/paper/2211.05025/) (2022)
- [Benchmarks for Pir'a 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change](https://mlsanity.com/paper/2309.10945/) (2023)
- [Deep Learning Brasil at ABSAPT 2022: Portuguese Transformer Ensemble Approaches](https://mlsanity.com/paper/2311.05051/) (2023)
- [FairytaleQA Translated: Enabling Educational Question and Answer Generation in Less-Resourced Languages](https://mlsanity.com/paper/2406.04233/) (2024)
- [The BLue Amazon Brain (BLAB): A Modular Architecture of Services about the Brazilian Maritime Territory](https://mlsanity.com/paper/2209.07928/) (2022)
- [BLAB Reporter: Automated journalism covering the Blue Amazon](https://mlsanity.com/paper/2210.06431/) (2022)

## Abstract (arXiv)

In natural language processing (NLP), there is a need for more resources in Portuguese, since much of the data used in the state-of-the-art research is in other languages. In this paper, we pretrain a T5 model on the BrWac corpus, an extensive collection of web pages in Portuguese, and evaluate its performance against other Portuguese pretrained models and multilingual models on three different tasks. We show that our Portuguese pretrained models have significantly better performance over the original T5 models. Moreover, we demonstrate the positive impact of using a Portuguese vocabulary. Our code and models are available at https://github.com/unicamp-dl/PTT5.