mlsanityData to 14 Oct 2026

Tucano: Advancing Neural Text Generation for Portuguese

arXiv 2411.07854 ↗First posted 12 Nov 2024cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Significant advances have been made in natural language processing in recent years. However, our current deep learning approach to language modeling requires substantial resources in terms of data and computation. One of the side effects of this data-hungry paradigm is the current schism between languages, separating those considered high-resource, where most of the development happens and resources are available, and the low-resource ones, which struggle to attain the same level of performance and autonomy. This study aims to introduce a new set of resources to stimulate the future development of neural text generation in Portuguese. In this work, we document the development of GigaVerbo, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens. Via this corpus, we trained a series of decoder-transformers named Tucano. Our models perform equal or superior to other Portuguese and multilingual language models of similar size in several Portuguese benchmarks. The evaluation of our models also reveals that model performance on many currently available benchmarks used by the Portuguese NLP community has little to no correlation with the scaling of token ingestion during training, highlighting the limitations of such evaluations when it comes to the assessment of Portuguese generative language models. All derivatives of our study are openly released on GitHub and Hugging Face. See https://nkluge-correa.github.io/Tucano/

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 116 of its 183 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

116 of 183 references.

References · 116

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  12. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  13. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  14. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  15. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  16. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  17. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  18. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  19. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  20. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  21. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  22. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  23. 2020GLU Variants Improve Transformercited by 1,804
  24. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  25. 2019Root Mean Square Layer Normalizationcited by 1,480
  26. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  27. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  28. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  29. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  30. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  31. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  32. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  33. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  34. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  35. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  36. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  37. 2024Yi: Open Foundation Models by 01.AIcited by 719
  38. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 10
  2. 2025Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corporacited by 7
  3. 2025PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguesecited by 5
  4. 2025Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretrainingcited by 4
  5. 2025BRoverbs -- Measuring how much LLMs understand Portuguese proverbscited by 3
  6. 2026NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpuscited by 1
  7. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  8. 2026Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMscited by 0
  9. 2026Language Models for Portuguese: A Systematic Mapping Studycited by 0
  10. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?cited by 0
  11. 2026Manac'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluationcited by 0
  12. 2026Break Step: Recursive Training Resonates with Replayed Sampling Noisecited by 0
1 more
  1. 2026Manac'a-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judgecited by 0