mlsanityData to 14 Oct 2026

Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench

arXiv 2601.02737 ↗First posted 6 Jan 2026cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in tasks such as abnormality detection and report generation for anatomical modalities, their capability in functional imaging remains largely unexplored. In this work, we identify and quantify a fundamental functional perception gap: the inability of current vision encoders to decode functional tracer biodistribution independent of morphological priors. Identifying Positron Emission Tomography (PET) as the quintessential modality to investigate this disconnect, we introduce PET-Bench, the first large-scale functional imaging benchmark comprising 52,308 hierarchical QA pairs from 9,732 multi-site, multi-tracer PET studies. Extensive evaluation of 19 state-of-the-art MLLMs reveals a critical safety hazard termed the Chain-of-Thought (CoT) hallucination trap. We observe that standard CoT prompting, widely considered to enhance reasoning, paradoxically decouples linguistic generation from visual evidence in PET, producing clinically fluent but factually ungrounded diagnoses. To resolve this, we propose Atomic Visual Alignment (AVA), a simple fine-tuning strategy that enforces the mastery of low-level functional perception prior to high-level diagnostic reasoning. Our results demonstrate that AVA effectively bridges the perception gap, transforming CoT from a source of hallucination into a robust inference tool and improving diagnostic accuracy by up to 14.83%. Code and data are available at https://github.com/yezanting/PET-Bench.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 64 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

38 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025OralGPT-Omni: A Versatile Dental Multimodal Large Language Model27 Nov
  2. 2025OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks2 Nov
  3. 2025S-Chain: Structured Visual Chain-of-Thought For Medicine26 Oct
  4. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation29 Sep
  5. 2025ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine20 Aug
  6. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review23 Jun
  7. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
  8. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?30 May
  9. 2025Multimodal Large Language Models for Medicine: A Comprehensive Survey29 Apr
  10. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models18 Mar
  11. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning26 Feb
  12. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation14 Feb
2 more
  1. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding24 Jan
  2. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI21 Nov

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  11. 2025MedGemma Technical Reportcited by 436
  12. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
12 more
  1. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  2. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  3. 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 102
  4. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  5. 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
  6. 2025Vision Foundation Models for Computed Tomographycited by 47
  7. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  8. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 17
  9. 2025LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanationscited by 16
  10. 2025V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosiscited by 15
  11. 2025PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomographycited by 7
  12. 2025Self is the Best Learner: CT-free Ultra-Low-Dose PET Organ Segmentation via Collaborating Denoising and Segmentation Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence
  2. 2026MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
  3. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
  4. 2026Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
  5. 2026Can Multimodal Large Language Models Understand OCT?
  6. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
  7. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions
  8. 2026Medical Image Alignment Assessment as a Test of Generalist Visual Reasoning in Frontier Multimodal Models