# Juru: Legal Brazilian Large Language Model from Reputable Sources

arXiv: https://arxiv.org/abs/2403.18140 · HTML: https://mlsanity.com/paper/2403.18140/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Cabrita: closing the gap for foreign languages](https://mlsanity.com/paper/2308.11878/) (2023)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)
- [BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams](https://mlsanity.com/paper/2307.05410/) (2023)
- [Sabi'a-2: A New Generation of Portuguese Large Language Models](https://mlsanity.com/paper/2403.09887/) (2024)
- [LegalNLP -- Natural Language Processing methods for the Brazilian Legal Language](https://mlsanity.com/paper/2110.15709/) (2021)
- [Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams](https://mlsanity.com/paper/2311.14169/) (2023)

## Built on by (most cited first, 5 of 5)

- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [Sabi'a-4 Technical Report](https://mlsanity.com/paper/2603.10213/) (2026)
- [ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine](https://mlsanity.com/paper/2412.14771/) (2024)
- [Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent Systems](https://mlsanity.com/paper/2608.09964/) (2026)
- [Language Models for Portuguese: A Systematic Mapping Study](https://mlsanity.com/paper/2608.18138/) (2026)

## Abstract (arXiv)

The high compute cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we specialized the Mistral-7B model with 1.9 billion unique tokens from reputable Brazilian legal sources and conducted few-shot evaluations on legal and general knowledge test suites. Our model, Juru, demonstrates the benefits of domain specialization by achieving improved performance on legal benchmarks, even with a reduced amount of pretraining data. However, this domain specialization through continued pretraining comes at the cost of increased forgetting in unrelated domains, as evidenced by performance degradation on general knowledge test suites in both Portuguese and English. This study contributes to the growing body of scientific evidence showing that pretraining data selection may enhance the performance of large language models, enabling the exploration of these models at a lower cost. Juru is publicly available at https://huggingface.co/roseval/Juru-7B .