# BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

arXiv: https://arxiv.org/abs/2508.21294 · HTML: https://mlsanity.com/paper/2508.21294/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 29 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension](https://mlsanity.com/paper/2404.16790/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [Internet-augmented language models through few-shot prompting for open-domain question answering](https://mlsanity.com/paper/2203.05115/) (2022)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion](https://mlsanity.com/paper/2306.11593/) (2023)
- [BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams](https://mlsanity.com/paper/2307.05410/) (2023)
- [TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time](https://mlsanity.com/paper/2501.07482/) (2025)
- [Passing the Brazilian OAB Exam: data preparation and some experiments](https://mlsanity.com/paper/1712.05128/) (2017)
- [Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams](https://mlsanity.com/paper/2311.14169/) (2023)

## Built on by (most cited first, 3 of 3)

- [Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining](https://mlsanity.com/paper/2512.12770/) (2025)
- [MARCA: A Checklist-Based Benchmark for Multilingual Web Search](https://mlsanity.com/paper/2604.14448/) (2026)
- [BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams](https://mlsanity.com/paper/2606.22723/) (2026)

## Abstract (arXiv)

With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models, enhancing its relevance for data contamination studies in LLM pretraining. Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions, more than doubling the number in the original BLUEX. We evaluated commercial and open-source LLMs and their ability to leverage visual context through captions.