# Noise-Aware Visual Representation Learning for Medical Visual Question Answering

arXiv: https://arxiv.org/abs/2606.05535 · HTML: https://mlsanity.com/paper/2606.05535/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Medical image denoising using convolutional denoising autoencoders](https://mlsanity.com/paper/1608.04667/) (2016)
- [Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models](https://mlsanity.com/paper/2303.05977/) (2023)
- [MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts](https://mlsanity.com/paper/2305.10799/) (2023)
- [Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models](https://mlsanity.com/paper/2404.10237/) (2024)
- [The role of noise in denoising models for anomaly detection in medical images](https://mlsanity.com/paper/2301.08330/) (2023)
- [PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging](https://mlsanity.com/paper/2401.02797/) (2024)
- [Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering](https://mlsanity.com/paper/2404.16192/) (2024)
- [A task-specific deep-learning-based denoising approach for myocardial perfusion SPECT](https://mlsanity.com/paper/2303.00212/) (2023)

## Abstract (arXiv)

Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf vision encoders with large language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting embeddings are then projected into the language model embedding space using a multi-layer perceptron (MLP), forming visual prefix tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA benchmarks. Experimental results show improved robustness to noisy input embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.