# P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

arXiv: https://arxiv.org/abs/2606.16753 · HTML: https://mlsanity.com/paper/2606.16753/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT*](https://mlsanity.com/paper/2402.18766/) (2024)
- [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022)
- [Gl'orIA -- A Generative and Open Large Language Model for Portuguese](https://mlsanity.com/paper/2402.12969/) (2024)
- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [IberBench: LLM Evaluation on Iberian Languages](https://mlsanity.com/paper/2504.16921/) (2025)
- [Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks](https://mlsanity.com/paper/2510.00962/) (2025)
- [Enhancing Portuguese Variety Identification with Cross-Domain Approaches](https://mlsanity.com/paper/2502.14394/) (2025)
- [Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties](https://mlsanity.com/paper/2505.20875/) (2025)
- [Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese](https://mlsanity.com/paper/2410.10991/) (2024)

## Built on by (most cited first, 1 of 1)

- [Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web](https://mlsanity.com/paper/2609.07699/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.