mlsanityData to 14 Oct 2026

Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam

arXiv 2507.19885 ↗First posted 26 Jul 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) has shown the potential to revolutionize healthcare by improving diagnostic accuracy, optimizing workflows, and personalizing treatment plans. Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have achieved notable advancements in natural language processing and medical applications. However, the evaluation of these models has focused predominantly on the English language, leading to potential biases in their performance across different languages. This study investigates the capability of six LLMs (GPT-4.0 Turbo, LLaMA-3-8B, LLaMA-3-70B, Mixtral 8x7B Instruct, Titan Text G1-Express, and Command R+) and four MLLMs (Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Sonnet, and Claude-3-Haiku) to answer questions written in Brazilian spoken portuguese from the medical residency entrance exam of the Hospital das Cl'inicas da Faculdade de Medicina da Universidade de S~ao Paulo (HCFMUSP) - the largest health complex in South America. The performance of the models was benchmarked against human candidates, analyzing accuracy, processing time, and coherence of the generated explanations. The results show that while some models, particularly Claude-3.5-Sonnet and Claude-3-Opus, achieved accuracy levels comparable to human candidates, performance gaps persist, particularly in multimodal questions requiring image interpretation. Furthermore, the study highlights language disparities, emphasizing the need for further fine-tuning and data set augmentation for non-English medical AI applications. Our findings reinforce the importance of evaluating generative AI in various linguistic and clinical settings to ensure a fair and reliable deployment in healthcare. Future research should explore improved training methodologies, improved multimodal reasoning, and real-world clinical integration of AI-driven medical assistance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 47 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

11 of 47 references.

References · 11

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  4. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  5. 2017Multitask learning and benchmarking with clinical time series datacited by 290
  6. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
  7. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  8. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Contextcited by 20
  9. 2024Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Taskcited by 12
  10. 2024Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large Language Models with Machine Learning in tele-dermatologycited by 4
  11. 2022Better to be in agreement than in bad company: a critical analysis of many kappa-like tests assessing one-million 2x2 contingency tablescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Performance of Large Language Models in Supporting Medical Diagnosis and Treatment
  2. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  4. 2024Entry-level guide to the use of large language models for medical research
  5. 2026EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation
  6. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
  7. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
  8. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?