# BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams

arXiv: https://arxiv.org/abs/2307.05410 · HTML: https://mlsanity.com/paper/2307.05410/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling](https://mlsanity.com/paper/2207.06814/) (2022)
- [A Review of Winograd Schema Challenge Datasets and Approaches](https://mlsanity.com/paper/2004.13831/) (2020)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Building a Sentiment Corpus of Tweets in Brazilian Portuguese](https://mlsanity.com/paper/1712.08917/) (2017)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)

## Built on by (most cited first, 15 of 15)

- [Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT*](https://mlsanity.com/paper/2402.18766/) (2024)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [Sabi'a-2: A New Generation of Portuguese Large Language Models](https://mlsanity.com/paper/2403.09887/) (2024)
- [Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora](https://mlsanity.com/paper/2509.08824/) (2025)
- [Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams](https://mlsanity.com/paper/2311.14169/) (2023)
- [Juru: Legal Brazilian Large Language Model from Reputable Sources](https://mlsanity.com/paper/2403.18140/) (2024)
- [PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese](https://mlsanity.com/paper/2511.17808/) (2025)
- [BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning](https://mlsanity.com/paper/2508.21294/) (2025)
- [BRoverbs -- Measuring how much LLMs understand Portuguese proverbs](https://mlsanity.com/paper/2509.08960/) (2025)
- [Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?](https://mlsanity.com/paper/2508.15835/) (2025)
- [MARCA: A Checklist-Based Benchmark for Multilingual Web Search](https://mlsanity.com/paper/2604.14448/) (2026)
- [Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines](https://mlsanity.com/paper/2605.01077/) (2026)
- [Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese](https://mlsanity.com/paper/2605.01630/) (2026)
- [BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams](https://mlsanity.com/paper/2606.22723/) (2026)
- [Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent Systems](https://mlsanity.com/paper/2608.09964/) (2026)

## Abstract (arXiv)

One common trend in recent studies of language models (LMs) is the use of standardized tests for evaluation. However, despite being the fifth most spoken language worldwide, few such evaluations have been conducted in Portuguese. This is mainly due to the lack of high-quality datasets available to the community for carrying out evaluations in Portuguese. To address this gap, we introduce the Brazilian Leading Universities Entrance eXams (BLUEX), a dataset of entrance exams from the two leading universities in Brazil: UNICAMP and USP. The dataset includes annotated metadata for evaluating the performance of NLP models on a variety of subjects. Furthermore, BLUEX includes a collection of recently administered exams that are unlikely to be included in the training data of many popular LMs as of 2023. The dataset is also annotated to indicate the position of images in each question, providing a valuable resource for advancing the state-of-the-art in multimodal language understanding and reasoning. We describe the creation and characteristics of BLUEX and establish a benchmark through experiments with state-of-the-art LMs, demonstrating its potential for advancing the state-of-the-art in natural language understanding and reasoning in Portuguese. The data and relevant code can be found at https://github.com/Portuguese-Benchmark-Datasets/BLUEX