mlsanityData to 14 Oct 2026

S2COPE: Self-Supervised Concept Discovery via Preference Learning

arXiv 2606.14586 ↗First posted 12 Jun 2026cs.CVcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Self-supervised learning can learn useful representations from unlabeled images. Yet these representations remain largely opaque, making it difficult to understand what visual concepts they encode. Can self-supervision instead produce useful visual concepts that are directly understandable to people? We introduce S2COPE, a framework that represents images as sets of concepts expressed in natural language. A vision-language model proposes candidate concepts, and a self-supervised objective evaluates how well each concept describes the source image while distinguishing it from other images. This feedback is used to construct preferences and train the model to propose better concepts. Across eight diverse visual datasets, our self-supervised concept representations transfer without adaptation and improve downstream recognition accuracy over prior interpretable approaches. In medical imaging, our discovered concepts reveal spurious visual cues and enable a more robust classifier. These results show that self-supervision can discover useful, human-readable visual representations from unlabeled images.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 72 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

56 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning8 Apr
  2. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
  3. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization17 Nov
  4. 2024Self-Supervised Visual Preference Alignment16 Apr

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  9. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  12. 2020Concept Bottleneck Modelscited by 1,032
  13. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  14. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  15. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  16. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  17. 2018Object Hallucination in Image Captioningcited by 669
  18. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  19. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  20. 2022Visual Classification via Description from Large Language Modelscited by 336
  21. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  22. 2022Post-hoc Concept Bottleneck Modelscited by 293
  23. 2023Label-Free Concept Bottleneck Modelscited by 285
  24. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  25. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  26. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  27. 2019Interpretations are useful: penalizing explanations to align neural networks with prior knowledgecited by 145
  28. 2018Discovering physical concepts with neural networkscited by 117
  29. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  30. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  31. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  32. 2024Reconstructive Visual Instruction Tuningcited by 55
  33. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  34. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  35. 2025Visual Jigsaw Post-Training Improves MLLMscited by 26
  36. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  37. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  38. 2025There Was Never a Bottleneck in Concept Bottleneck Modelscited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Concept Bottleneck Models Without Predefined Concepts
  2. 2024Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations
  3. 2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use
  4. 2025CUBIC: Concept Embeddings for Unsupervised Bias Identification using VLMs
  5. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery
  6. 2025ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models
  7. 2023Coarse-to-Fine Concept Bottleneck Models
  8. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?