mlsanityData to 7 Oct 2026

SAM2CLIP2SAM: Vision Language Model for Segmentation of 3D CT Scans for Covid-19 Detection

arXiv 2407.15728 ↗First posted 22 Jul 2024eess.IV cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

This paper presents a new approach for effective segmentation of images that can be integrated into any model and methodology; the paradigm that we choose is classification of medical images (3-D chest CT scans) for Covid-19 detection. Our approach includes a combination of vision-language models that segment the CT scans, which are then fed to a deep neural architecture, named RACNet, for Covid-19 detection. In particular, a novel framework, named SAM2CLIP2SAM, is introduced for segmentation that leverages the strengths of both Segment Anything Model (SAM) and Contrastive Language-Image Pre-Training (CLIP) to accurately segment the right and left lungs in CT scans, subsequently feeding these segmented outputs into RACNet for classification of COVID-19 and non-COVID-19 cases. At first, SAM produces multiple part-based segmentation masks for each slice in the CT scan; then CLIP selects only the masks that are associated with the regions of interest (ROIs), i.e., the right and left lungs; finally SAM is given these ROIs as prompts and generates the final segmentation mask for the lungs. Experiments are presented across two Covid-19 annotated databases which illustrate the improved performance obtained when our method has been used for segmentation of the CT scans.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

36 of 62 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Segment Anythingcited by 8,158
  4. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  5. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  6. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  7. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  8. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  9. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  10. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
  11. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  12. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
24 more
  1. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
  2. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  3. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  4. 2019Bi-Directional ConvLSTM U-Net with Densley Connected Convolutionscited by 82
  5. 2020Deep Transparent Prediction through Latent Representation Analysiscited by 72
  6. 2021MIA-COV19D: COVID-19 Detection through 3-D Chest CT Image Analysiscited by 72
  7. 2024Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyondcited by 60
  8. 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53
  9. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
  10. 2023A Deep Neural Architecture for Harmonizing 3-D Input Data Analysis and Decision Making in Medical Imagingcited by 39
  11. 2022MixAugment & Mixup: Augmentation Methods for Facial Expression Recognitioncited by 31
  12. 2024Domain adaptation, Explainability & Fairness in AI for Medical Image Analysis: Diagnosis of COVID-19 based on 3-D Chest CT-scanscited by 30
  13. 2017Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classificationcited by 27
  14. 2024COVID-19 Computer-aided Diagnosis through AI-assisted CT Imaging Analysis: Deploying a Medical AI Systemcited by 23
  15. 2023BTDNet: a Multi-Modal Approach for Brain Tumor Radiogenomic Classificationcited by 20
  16. 2024Uncertainty-guided Contrastive Learning for Single Source Domain Generalisationcited by 14
  17. 2021Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scanscited by 13
  18. 2024Ensuring UAV Safety: A Vision-only and Real-time Framework for Collision Avoidance Through Object Detection, Tracking, and Distance Estimationcited by 11
  19. 2024Can Machine Learning Assist in Diagnosis of Primary Immune Thrombocytopenia? A feasibility studycited by 8
  20. 2022Spatiotemporal Feature Learning Based on Two-Step LSTM and Transformer for CT Scanscited by 7
  21. 2021Evolutionary Multi-objective Architecture Search Framework: Application to COVID-19 3D CT Classificationcited by 5
  22. 2022FDVTS's Solution for 2nd COV19D Competition on COVID-19 Detection and Severity Analysiscited by 4
  23. 2022Cov3d: Detection of the presence and severity of COVID-19 from CT scans using 3D ResNetscited by 4
  24. 2023COVID-19 Detection Using Slices Processing Techniques and a Modified Xception Classifier from Computed Tomography Imagescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 6.

14 papers.

Sort:

  1. 2024Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkitcited by 16
  2. 2025DVD: A Comprehensive Dataset for Advancing Violence Detection in Real-World Scenarioscited by 13
  3. 2024Using Prompts to Guide Large Language Models in Imitating a Real Person's Language Stylecited by 6
  4. 2025Multi-Source COVID-19 Detection via Variance Risk Extrapolationcited by 5
  5. 2025Advancing Lung Disease Diagnosis in 3D CT Scanscited by 4
  6. 2025Taming Domain Shift in Multi-source CT-Scan Classification via Input-Space Standardizationcited by 2
  7. 2025Multi Source COVID-19 Detection via Kernel-Density-based Slice Samplingcited by 1
  8. 2025SAM-PTx: Text-Guided Fine-Tuning of SAM with Parameter-Efficient, Parallel-Text Adapterscited by 1
  9. 2026A Heterogeneous Ensemble for Multi-Center COVID-19 Classification from Chest CT Scanscited by 0
  10. 2026Halfway to 3D: Ensembling 2.5D and 3D Models for Robust COVID-19 CT Diagnosiscited by 0
  11. 2026Clinical Priors Guided Lung Disease Detection in 3D CT Scanscited by 0
  12. 2026Vision-Language Model Based Multi-Expert Fusion for CT Image Classificationcited by 0
2 more
  1. 2026Towards Fair and Robust Volumetric CT Classification via KL-Regularised Group Distributionally Robust Optimisationcited by 0
  2. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis
  2. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
  3. 2025STPNet: Scale-aware Text Prompt Network for Medical Image Segmentation
  4. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
  5. 2025Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation
  6. 2025Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein
  7. 2025Unified Supervision For Vision-Language Modeling in 3D Computed Tomography
  8. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT