mlsanityData to 14 Oct 2026

Skin Lesion Phenotyping via Nested Multi-modal Contrastive Learning

arXiv 2505.23709 ↗First posted 29 May 2025cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce SLIMP (Skin Lesion Image-Metadata Pre-training) for learning rich representations of skin lesions through a novel nested contrastive learning approach that captures complex relationships between images and metadata. Melanoma detection and skin lesion classification based solely on images, pose significant challenges due to large variations in imaging conditions (lighting, color, resolution, distance, etc.) and lack of clinical and phenotypical context. Clinicians typically follow a holistic approach for assessing the risk level of the patient and for deciding which lesions may be malignant and need to be excised, by considering the patient's medical history as well as the appearance of other lesions of the patient. Inspired by this, SLIMP combines the appearance and the metadata of individual skin lesions with patient-level metadata relating to their medical record and other clinically relevant information. By fully exploiting all available data modalities throughout the learning process, the proposed pre-training strategy improves performance compared to other pre-training strategies on downstream skin lesions classification tasks highlighting the learned representations quality.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

36 of 47 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
24 more
  1. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  2. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  3. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  4. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  5. 2020PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphonescited by 118
  6. 2023Towards Geospatial Foundation Models via Continual Pretrainingcited by 109
  7. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 82
  8. 2022Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Frameworkcited by 59
  9. 2023Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Seriescited by 59
  10. 2024A Multimodal Vision Foundation Model for Clinical Dermatologycited by 50
  11. 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
  12. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  13. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  14. 2023Subclass-balancing Contrastive Learning for Long-tailed Recognitioncited by 26
  15. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  16. 2022A survey, review, and future trends of skin lesion segmentation and classificationcited by 21
  17. 2022Discrete Key-Value Bottleneckcited by 18
  18. 2022TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosiscited by 15
  19. 2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
  20. 2023Hierarchical discriminative learning improves visual representations of biomedical microscopycited by 12
  21. 2019Multimodal Multitask Representation Learning for Pathology Biobank Metadata Predictioncited by 10
  22. 2023Towards Concept-based Interpretability of Skin Lesion Diagnosis using Vision-Language Modelscited by 9
  23. 2025Enhancing Vision Foundation Models via Multimodal Continual Pre-Trainingcited by 5
  24. 2024TRACE: Transformer-based Risk Assessment for Clinical Evaluationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inferencecited by 0
  2. 2026Geometry-Aware Superpixel Graph Transformer with Metadata for Skin Lesion Classificationcited by 0