mlsanityData to 14 Oct 2026

TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

arXiv 2401.16640 ↗First posted 30 Jan 2024cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 83 of its 112 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

83 of 112 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  15. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  16. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  17. 2020GLU Variants Improve Transformercited by 1,804
  18. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  19. 2019Root Mean Square Layer Normalizationcited by 1,480
  20. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  21. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  22. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  23. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  24. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  25. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  26. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  27. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  28. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  29. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  30. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  31. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  32. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  33. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  34. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  35. 2023Textbooks Are All You Needcited by 569
  36. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  37. 2023The Falcon Series of Open Language Modelscited by 535
  38. 2021Datasets: A Community Library for Natural Language Processingcited by 530

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 13
  2. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  3. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  4. 2025PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguesecited by 5
  5. 2025Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretrainingcited by 4
  6. 2024SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclaturecited by 3
  7. 2024From Intents to Conversations: Generating Intent-Driven Dialogues with Contrastive Learning for Multi-Turn Classificationcited by 2
  8. 2026Soro: A Lightweight Foundation Model and Chatbot for Tajikcited by 2
  9. 2024Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the S'ami Languagecited by 1
  10. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  11. 2026Language Models for Portuguese: A Systematic Mapping Studycited by 0
  12. 2026Manac'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluationcited by 0
1 more
  1. 2026Manac'a-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judgecited by 0