mlsanityData to 14 Oct 2026

Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision

arXiv 2509.12771 ↗First posted 16 Sep 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans can recognize an image as an instance of a general concept, beyond simply identifying its objects and their relationships. In this paper, we investigate 1. The extent to which VLMs have this concept abstraction capacity, and 2. Strategies for encoding the sort of higher-concept information in images that would enable the resulting VLM model (CLEAR GLASS model) to have this capability to a greater degree. To this end, we introduce a grouped image-caption dataset (MAGIC), which consists of several groups of image captions and for each group a set of associated images and higher-level conceptual labels. We use a novel contrastive loss technique to induce the model to encode in the representation of each image (caption) in a group the information that is common to all members of the image-caption group. Our main contribution is a grouped contrastive loss function based on text-image contrastive groups (outer contrastive loss) as well as an inner loss which measures the distances between image-caption instances in the group. Our training methodology results in the CLEAR GLASS model having the concept abstraction capacity as an emergent capacity because the model is not exposed to the higher-level concepts associated with each group. Instead, the training forces the model to create for each image-caption group a semantic representation that brings it closer to the semantic representation of the higher-level concepts in the latent semantic space. Our experiments show that this training methodology results in a model which shows improvement in abstract concept recognition compared to SOTA models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

42 of 58 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  12. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
30 more
  1. 2022Generalized Category Discoverycited by 211
  2. 2024An Introduction to Vision-Language Modelingcited by 178
  3. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  4. 2023Hyperbolic Image-Text Representationscited by 136
  5. 2017Representation Learning by Learning to Countcited by 120
  6. 2022Parametric Classification for Generalized Category Discovery: A Baseline Studycited by 113
  7. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 101
  8. 2023Dynamic Conceptional Contrastive Learning for Generalized Category Discoverycited by 92
  9. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  10. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 84
  11. 2020Hard negative examples are hard, but usefulcited by 82
  12. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  13. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  14. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  15. 2024Active Generalized Category Discoverycited by 31
  16. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 31
  17. 2023Federated Generalized Category Discoverycited by 27
  18. 2024Open-world machine learning: A review and new outlookscited by 27
  19. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 27
  20. 2023CLIP-GCD: Simple Language Guided Generalized Category Discoverycited by 25
  21. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  22. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  23. 2024Pre-trained Vision-Language Models Learn Discoverable Visual Conceptscited by 16
  24. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  25. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  26. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  27. 2022Beyond Object Recognition: A New Benchmark towards Object Concept Learningcited by 11
  28. 2023GeNIe: Generative Hard Negative Images Through Diffusioncited by 8
  29. 2024Multimodal Generalized Category Discoverycited by 7
  30. 2016Learning Concept Taxonomies from Multi-modal Datacited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
  2. 2023Image Captioners Are Scalable Vision Learners Too
  3. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
  4. 2022Perceptual Grouping in Contrastive Vision-Language Models
  5. 2025Emergence of Text Readability in Vision Language Models
  6. 2022Unifying Vision-Language Representation Space with Single-tower Transformer
  7. 2025Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
  8. 2024DreamLIP: Language-Image Pre-training with Long Captions