# How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking

arXiv: https://arxiv.org/abs/2605.18111 · HTML: https://mlsanity.com/paper/2605.18111/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [MedICaT: A Dataset of Medical Images, Captions, and Textual References](https://mlsanity.com/paper/2010.06000/) (2020)
- [Where To Look: Focus Regions for Visual Question Answering](https://mlsanity.com/paper/1511.07394/) (2015)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA](https://mlsanity.com/paper/2405.20421/) (2024)
- [ChitroJera: A Regionally Relevant Visual Question Answering Dataset for Bangla](https://mlsanity.com/paper/2410.14991/) (2024)
- [Free Form Medical Visual Question Answering in Radiology](https://mlsanity.com/paper/2401.13081/) (2024)
- [Bangla-Bayanno: A 52K-Pair Bengali Visual Question Answering Dataset with LLM-Assisted Translation Refinement](https://mlsanity.com/paper/2508.19887/) (2025)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) and Large Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promising capabilities in complex reasoning tasks, including those in the medical domain. Medical Visual Question Answering (MedVQA) has particularly benefited from these developments. However, despite Bangla being one of the most widely spoken languages globally, there exists no established MedVQA benchmark for it. To address this gap, we introduce BanglaMedVQA, a dataset comprising clinically validated image-question-answer pairs, along with a comprehensive evaluation of current foundation models on this resource. Consistent with prior findings that report low performance of current models on English MedVQA benchmarks, our analysis reveals that Bangla performance is substantially lower, reflecting the challenges inherent to low-resource languages. Even top-performing models such as Gemini and GPT-4.1 mini fail to accurately answer specialized diagnostic questions, indicating severe limitations in fine-grained medical reasoning. Although certain open-source models, such as Gemma-3, occasionally outperform these models in general categories, they too struggle with clinically complex questions, underscoring the urgent need for top-notch evaluation method.