mlsanityData to 7 Oct 2026

MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

arXiv 2603.17325 ↗First posted 18 Mar 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention(TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

36 of 51 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 614
  10. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 507
  11. 2022Multimodal Learning with Transformers: A Surveycited by 399
  12. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
24 more
  1. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  2. 2022Diffusion Models for Medical Anomaly Detectioncited by 231
  3. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  4. 2020Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Imagescited by 184
  5. 2024AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detectioncited by 126
  6. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  7. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  8. 2022Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detectioncited by 104
  9. 2021Mean-Shifted Contrastive Loss for Anomaly Detectioncited by 87
  10. 2023BMAD: Benchmarks for Medical Anomaly Detectioncited by 81
  11. 2022Explicit Boundary Guided Semi-Push-Pull Contrastive Learning for Supervised Anomaly Detectioncited by 72
  12. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  13. 2019Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practicecited by 71
  14. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  15. 2018Context-encoding Variational Autoencoder for Unsupervised Anomaly Detectioncited by 63
  16. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  17. 2020Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Imagescited by 52
  18. 2020MADGAN: unsupervised Medical Anomaly Detection GAN using multiple adjacent brain MRI slice reconstructioncited by 50
  19. 2019Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCTcited by 41
  20. 2023CLIP in Medical Imaging: A Surveycited by 40
  21. 2024MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detectioncited by 35
  22. 2021SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detectioncited by 34
  23. 2021Max-Margin Contrastive Learningcited by 27
  24. 2025MadCLIP: Few-shot Medical Anomaly Detection with CLIPcited by 10

Built on by

Not cited yet.