# HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

arXiv: https://arxiv.org/abs/2506.21578 · HTML: https://mlsanity.com/paper/2506.21578/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 3 of 3)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset](https://mlsanity.com/paper/2411.15640/) (2024)
- [Trust and Medical AI: The challenges we face and the expertise needed to overcome them](https://mlsanity.com/paper/2008.07734/) (2020)

## Built on by (most cited first, 3 of 3)

- [PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams -- Dataset Construction and Evaluation](https://mlsanity.com/paper/2509.11517/) (2025)
- [Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines](https://mlsanity.com/paper/2605.01077/) (2026)
- [Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering](https://mlsanity.com/paper/2607.06641/) (2026)

## Abstract (arXiv)

The evaluation of Large Language Models (LLMs) in healthcare has been dominated by physician-centric, English-language benchmarks, creating a dangerous illusion of competence that ignores the interprofessional nature of patient care. To provide a more holistic and realistic assessment, we introduce HealthQA-BR, the first large-scale, system-wide benchmark for Portuguese-speaking healthcare. Comprising 5,632 questions from Brazil's national licensing and residency exams, it uniquely assesses knowledge not only in medicine and its specialties but also in nursing, dentistry, psychology, social work, and other allied health professions. We conducted a rigorous zero-shot evaluation of over 20 leading LLMs. Our results reveal that while state-of-the-art models like GPT 4.1 achieve high overall accuracy (86.6%), this top-line score masks alarming, previously unmeasured deficiencies. A granular analysis shows performance plummets from near-perfect in specialties like Ophthalmology (98.7%) to barely passing in Neurosurgery (60.0%) and, most notably, Social Work (68.4%). This "spiky" knowledge profile is a systemic issue observed across all models, demonstrating that high-level scores are insufficient for safety validation. By publicly releasing HealthQA-BR and our evaluation suite, we provide a crucial tool to move beyond single-score evaluations and toward a more honest, granular audit of AI readiness for the entire healthcare team.