# PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese

arXiv: https://arxiv.org/abs/2511.17808 · HTML: https://mlsanity.com/paper/2511.17808/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Builds on (most cited first, 30 of 37)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [Multiple-Choice Questions are Efficient and Robust LLM Evaluators](https://mlsanity.com/paper/2405.11966/) (2024)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Cabrita: closing the gap for foreign languages](https://mlsanity.com/paper/2308.11878/) (2023)
- [Building a Sentiment Corpus of Tweets in Brazilian Portuguese](https://mlsanity.com/paper/1712.08917/) (2017)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)
- [Gl'orIA -- A Generative and Open Large Language Model for Portuguese](https://mlsanity.com/paper/2402.12969/) (2024)
- [HateBR: A Large Expert Annotated Corpus of Brazilian Instagram Comments for Offensive Language and Hate Speech Detection](https://mlsanity.com/paper/2103.14972/) (2021)
- [BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams](https://mlsanity.com/paper/2307.05410/) (2023)
- [TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese](https://mlsanity.com/paper/2401.16640/) (2024)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)

## Built on by (most cited first, 5 of 5)

- [Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining](https://mlsanity.com/paper/2512.12770/) (2025)
- [CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context](https://mlsanity.com/paper/2603.22576/) (2026)
- [Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining](https://mlsanity.com/paper/2603.24826/) (2026)
- [MARCA: A Checklist-Based Benchmark for Multilingual Web Search](https://mlsanity.com/paper/2604.14448/) (2026)
- [Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese](https://mlsanity.com/paper/2605.01630/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.