mlsanityData to 7 Oct 2026

Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning

arXiv 2505.18503 ↗First posted 24 May 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Large Vision-Language Models (Med-LVLMs) often exhibit suboptimal attention distribution on visual inputs, leading to hallucinated or inaccurate outputs. Existing mitigation methods primarily rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. To address this, we propose A3Tune, a novel fine-tuning framework for Automatic Attention Alignment Tuning. A3Tune leverages zero-shot weak labels from SAM, refines them into prompt-aware labels using BioMedCLIP, and then selectively modifies visually-critical attention heads to improve alignment while minimizing interference. Additionally, we introduce a A3MoE module, enabling adaptive parameter selection for attention tuning across diverse prompts and images. Extensive experiments on medical VQA and report generation benchmarks show that A3Tune outperforms state-of-the-art baselines, achieving enhanced attention distributions and performance in Med-LVLMs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 45 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

28 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models10 Jun
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  9. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  11. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
13 more
  1. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  2. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  3. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  4. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  5. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  6. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  7. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  8. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  9. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Contextcited by 19
  10. 2025TruthFlow: Truthful LLM Generation via Representation Flow Correctioncited by 14
  11. 2025MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Modelscited by 10
  12. 2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decodingcited by 8
  13. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decodingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Enhancing Medical Large Vision-Language Models via Alignment Distillationcited by 1
  2. 2025CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imagingcited by 0
  3. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
  2. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  3. 2026Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understanding
  4. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
  5. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
  6. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
  7. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning
  8. 2024Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models