mlsanityData to 7 Oct 2026

Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*

arXiv 2305.06721 ↗First posted 11 May 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

To advance the neural encoding of Portuguese (PT), and a fortiori the technological preparation of this language for the digital age, we developed a Transformer-based foundation model that sets a new state of the art in this respect for two of its variants, namely European Portuguese from Portugal (PT-PT) and American Portuguese from Brazil (PT-BR). To develop this encoder, which we named Albertina PT-*, a strong model was used as a starting point, DeBERTa, and its pre-training was done over data sets of Portuguese, namely over data sets we gathered for PT-PT and PT-BR, and over the brWaC corpus for PT-BR. The performance of Albertina and competing models was assessed by evaluating them on prominent downstream language processing tasks adapted for Portuguese. Both Albertina PT-PT and PT-BR versions are distributed free of charge and under the most permissive license possible and can be run on consumer-grade hardware, thus seeking to contribute to the advancement of research and innovation in language technology for Portuguese.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

20 of 36 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
8 more
  1. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  2. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  3. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  4. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  5. 2019Multilingual is not enough: BERT for Finnishcited by 118
  6. 2019BERTje: A Dutch BERT Modelcited by 109
  7. 2021MarIA: Spanish Language Modelscited by 36
  8. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 4; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT*cited by 19
  2. 2024Gl'orIA -- A Generative and Open Large Language Model for Portuguesecited by 16
  3. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  4. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  5. 2024Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Familycited by 8
  6. 2024ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Languagecited by 8
  7. 2025Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corporacited by 7
  8. 2024PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguesecited by 5
  9. 2024Open Sentence Embeddings for Portuguese with the Serafim PT* encoders familycited by 4
  10. 2025Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretrainingcited by 4
  11. 2023DeBERTinha: A Multistep Approach to Adapt DebertaV3 XSmall for Brazilian Portuguese Natural Language Processing Taskcited by 2
  12. 2024PeLLE: Encoder-based language models for Brazilian Portuguese based on open datacited by 1
12 more
  1. 2024A Legal Framework for Natural Language Processing Model Training in Portugalcited by 1
  2. 2024Meta-prompting Optimized Retrieval-augmented Generationcited by 1
  3. 2026NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpuscited by 1
  4. 2024MultiLS: A Multi-task Lexical Simplification Frameworkcited by 0
  5. 2025From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concernscited by 0
  6. 2025Uncovering Latent Connections in Indigenous Heritage: Semantic Pipelines for Cultural Preservation in Brazilcited by 0
  7. 2026CitiLink-Minutes: A Multilayer Annotated Dataset of Municipal Meeting Minutescited by 0
  8. 2026MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptationcited by 0
  9. 2026LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classificationcited by 0
  10. 2026moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTcited by 0
  11. 2026BERTomelo: Your Portuguese Encoder Best Friendcited by 0
  12. 2026Language Models for Portuguese: A Systematic Mapping Studycited by 0