# Beyond the Leaderboard: Design Lessons for Trustworthy Multimodal VQA

arXiv: https://arxiv.org/abs/2607.15241 · HTML: https://mlsanity.com/paper/2607.15241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://mlsanity.com/paper/2310.16836/) (2023)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [Kvasir-VQA: A Text-Image Pair GI Tract Dataset](https://mlsanity.com/paper/2409.01437/) (2024)
- [Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2506.09958/) (2025)
- [Medico 2025: Visual Question Answering for Gastrointestinal Imaging](https://mlsanity.com/paper/2508.10869/) (2025)
- [Multi-Task Learning for Visually Grounded Reasoning in Gastrointestinal VQA](https://mlsanity.com/paper/2511.04384/) (2025)

## Abstract (arXiv)

Healthcare multimodal AI must combine visual and textual evidence while remaining reliable and interpretable. Using MediaEval Medico 2025 as a retrospective GI endoscopy case study, we analyze design choices across nine documented systems for question answering and explanation quality. Parameter-efficient adaptation of pretrained backbones provides strong challenge performance, but answer-level gains do not consistently translate into faithful and complete clinical reasoning. Methods enforcing structured reasoning and explicit grounding show more reliable behavior across heterogeneous question types, although the evidence is correlational rather than ablation-based. These results motivate evaluation beyond lexical overlap, standardized evidence-linked explanations, leakage-aware data governance, and lightweight robustness and calibration checks. The findings support trustworthy multimodal healthcare AI based on data fusion, explainability, and resilient evaluation.