mlsanityData to 14 Oct 2026

BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

arXiv 2508.21294 ↗First posted 29 Aug 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models, enhancing its relevance for data contamination studies in LLM pretraining. Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions, more than doubling the number in the original BLUEX. We evaluated commercial and open-source LLMs and their ability to leverage visual context through captions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

26 of 35 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 445
14 more
  1. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  2. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  4. 2023Using an LLM to Help With Code Understandingcited by 228
  5. 2023Planning with Large Language Models for Code Generationcited by 182
  6. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  7. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  8. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  9. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  10. 2023Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusioncited by 21
  11. 2023BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXamscited by 15
  12. 2025TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Timecited by 8
  13. 2017Passing the Brazilian OAB Exam: data preparation and some experimentscited by 6
  14. 2023Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Examscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretrainingcited by 4
  2. 2026MARCA: A Checklist-Based Benchmark for Multilingual Web Searchcited by 0
  3. 2026BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Examscited by 0