mlsanityData to 14 Oct 2026

Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

arXiv 2403.12570 ↗First posted 19 Mar 2024cs.CVcited by 69

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 45 of its 69 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

45 of 69 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  7. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  9. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 776
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  12. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
33 more
  1. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  2. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 531
  3. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 518
  4. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 509
  5. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  6. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 421
  7. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  8. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  9. 2021CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flowscited by 321
  10. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  11. 2019Deep Semi-Supervised Anomaly Detectioncited by 291
  12. 2020Multiresolution Knowledge Distillation for Anomaly Detectioncited by 285
  13. 2022Diffusion Models for Medical Anomaly Detectioncited by 232
  14. 2018Adversarially Learned One-Class Classifier for Novelty Detectioncited by 202
  15. 2019Deep Anomaly Detection with Deviation Networkscited by 153
  16. 2022Registration based Few-Shot Anomaly Detectioncited by 133
  17. 2023APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot ADcited by 130
  18. 2020Deep Learning for Medical Anomaly Detection -- A Surveycited by 127
  19. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  20. 2022Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detectioncited by 104
  21. 2023Diffusion Model as Representation Learnercited by 95
  22. 2023BMAD: Benchmarks for Medical Anomaly Detectioncited by 82
  23. 2022Explicit Boundary Guided Semi-Push-Pull Contrastive Learning for Supervised Anomaly Detectioncited by 72
  24. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  25. 2021A Hierarchical Transformation-Discriminating Generative Model for Few Shot Anomaly Detectioncited by 60
  26. 2020Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Imagescited by 52
  27. 2021Proxy-bridged Image Reconstruction Network for Anomaly Detection in Medical Imagescited by 18
  28. 2023Mutual-modality Adversarial Attack with Semantic Perturbationcited by 14
  29. 2022AFSC: Adaptive Fourier Space Compression for Anomaly Detectioncited by 12
  30. 2021Unsupervised Anomaly Segmentation using Image-Semantic Cycle Translationcited by 11
  31. 2023Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly Detectioncited by 10
  32. 2023Multi-Scale Memory Comparison for Zero-/Few-Shot Anomaly Detectioncited by 3
  33. 2022LesionPaste: One-Shot Anomaly Detection for Medical Imagescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 44; 2026: 18.

69 papers.

Sort:

By area:

  1. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 73
  2. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  3. 2025Exploring Intrinsic Normal Prototypes within a Single Image for Universal Anomaly Detectioncited by 48
  4. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  5. 2024MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detectioncited by 35
  6. 2024UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detectioncited by 28
  7. 2024MedIAnomaly: A comparative study of anomaly detection in medical imagescited by 26
  8. 2024MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objectscited by 26
  9. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 26
  10. 2025Learning to Detect Multi-class Anomalies with Just One Normal Image Promptcited by 18
  11. 2025MadCLIP: Few-shot Medical Anomaly Detection with CLIPcited by 10
  12. 2025Normal-Abnormal Guided Generalist Anomaly Detectioncited by 10
38 more
  1. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  2. 2025FiLo++: Zero-/Few-Shot Anomaly Detection by Fused Fine-Grained Descriptions and Deformable Localizationcited by 8
  3. 2025AnomalyMoE: Towards a Language-free Generalist Model for Unified Visual Anomaly Detectioncited by 8
  4. 2025KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integrationcited by 7
  5. 2025One-to-Normal: Anomaly Personalization for Few-shot Anomaly Detectioncited by 7
  6. 2025INP-Former++: Advancing Universal Anomaly Detection via Intrinsic Normal Prototypes and Residual Learningcited by 7
  7. 2025Applications of Large Models in Medicinecited by 6
  8. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  9. 2025AnomalyPainter: Vision-Language-Diffusion Synergy for Zero-Shot Realistic and Diverse Industrial Anomaly Synthesiscited by 5
  10. 2025UltraAD: Fine-Grained Ultrasound Anomaly Classification via Few-Shot CLIP Adaptationcited by 5
  11. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Modelscited by 4
  12. 2025Continual-MEGA: A Large-scale Benchmark for Generalizable Continual Anomaly Detectioncited by 4
  13. 2025Towards Fine-Grained Vision-Language Alignment for Few-Shot Anomaly Detectioncited by 4
  14. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 4
  15. 2025Foundation Models for Anomaly Detection: Vision and Challengescited by 3
  16. 2025MoNetV2: Enhanced Motion Network for Freehand 3D Ultrasound Reconstructioncited by 3
  17. 2025An Evidence-Based Post-Hoc Adjustment Framework for Anomaly Detection Under Data Contaminationcited by 3
  18. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  19. 2025MuSc-V2: Zero-Shot Multimodal Industrial Anomaly Classification and Segmentation with Mutual Scoring of Unlabeled Samplescited by 3
  20. 2026Dual Distillation for Few-Shot Anomaly Detectioncited by 3
  21. 2024ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialoguecited by 2
  22. 2025MADPOT: Medical Anomaly Detection with CLIP Adaptation and Partial Optimal Transportcited by 2
  23. 2025A Comprehensive Survey for Real-World Industrial Defect Detection: Challenges, Approaches, and Prospectscited by 2
  24. 2025Q-Former Autoencoder: A Modern Framework for Medical Anomaly Detectioncited by 2
  25. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  26. 2025Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentationcited by 1
  27. 2025Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Modelscited by 1
  28. 2025IQE-CLIP: Instance-aware Query Embedding for Zero-/Few-shot Anomaly Detection in Medical Domaincited by 1
  29. 2025Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Scorecited by 1
  30. 2025Toward Long-Tailed Online Anomaly Detection through Class-Agnostic Conceptscited by 1
  31. 2025CoPS: Conditional Prompt Synthesis for Zero-Shot Anomaly Detectioncited by 1
  32. 2025COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasetscited by 1
  33. 2025A Contrastive Learning-Guided Confident Meta-learning for Zero Shot Anomaly Detectioncited by 1
  34. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancementcited by 1
  35. 2026Any2Any 3D Diffusion Models with Knowledge Transfer: A Radiotherapy Planning Studycited by 1
  36. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  37. 2024Two Is Better Than One: Aligned Representation Pairs for Anomaly Detectioncited by 0
  38. 2025When Unsupervised Domain Adaptation meets One-class Anomaly Detection: Addressing the Two-fold Unsupervised Curse by Leveraging Anomaly Scarcitycited by 0

50 most cited of 69 All 69 (JSON)