mlsanityData to 14 Oct 2026

Perceptual Grouping in Contrastive Vision-Language Models

arXiv 2210.09996 ↗First posted 18 Oct 2022cs.CV cs.LGcited by 82

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advances in zero-shot image recognition suggest that vision-language models learn generic visual representations with a high degree of semantic information that may be arbitrarily probed with natural language phrases. Understanding an image, however, is not just about understanding what content resides within an image, but importantly, where that content resides. In this work we examine how well vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery. We demonstrate how contemporary vision and language representation learning models based on contrastive losses and large web-based data capture limited object localization information. We propose a minimal set of modifications that results in models that uniquely learn both semantic and spatial information. We measure this performance in terms of zero-shot image recognition, unsupervised bottom-up and top-down semantic segmentations, as well as robustness analyses. We find that the resulting model achieves state-of-the-art results in terms of unsupervised segmentation, and demonstrate that the learned representations are uniquely robust to spurious correlations in datasets designed to probe the causal behavior of vision models.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 91 of its 137 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

91 of 137 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 88

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
38 more
  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  11. 2017Non-local Neural Networkscited by 2,293
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  13. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  14. 2019Invariant Risk Minimizationcited by 1,854
  15. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  16. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  18. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  19. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  21. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  23. 2021Grounded Language-Image Pre-trainingcited by 1,156
  24. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  26. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  27. 2020Object-Centric Learning with Slot Attentioncited by 876
  28. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  29. 2022Language-driven Semantic Segmentationcited by 717
  30. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  31. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  32. 2021Extract Free Dense Labels from CLIPcited by 583
  33. 2018Do Better ImageNet Models Transfer Better?cited by 565
  34. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  35. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  36. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  37. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  38. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 32; 2025: 21; 2026: 13.

82 papers.

Sort:

By area:

  1. 2023Data Filtering Networkscited by 250
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  3. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  4. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  5. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  6. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  7. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  8. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  9. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  10. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  11. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  12. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
38 more
  1. 2025Scaling Laws for Native Multimodal Modelscited by 50
  2. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 49
  3. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  4. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  5. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 43
  6. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  7. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  8. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  9. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  10. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  11. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  12. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  13. 2024HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Textscited by 17
  14. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  15. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  16. 2024DEAL: Disentangle and Localize Concept-level Explanations for VLMscited by 14
  17. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  18. 2024Is CLIP the main roadblock for fine-grained open-world perception?cited by 13
  19. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  20. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Modelscited by 12
  21. 2024Moving Off-the-Grid: Scene-Grounded Video Representationscited by 11
  22. 2023Language-based Action Concept Spaces Improve Video Self-Supervised Learningcited by 10
  23. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  24. 2025Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Modelscited by 10
  25. 2024PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformerscited by 9
  26. 2024Generalization Boosted Adapter for Open-Vocabulary Segmentationcited by 9
  27. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  28. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  29. 2024ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentationcited by 8
  30. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 8
  31. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  32. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  33. 2023Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generationcited by 6
  34. 2025SAB3R: Semantic-Augmented Backbone in 3D Reconstructioncited by 6
  35. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classificationcited by 5
  36. 2023Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentationcited by 4
  37. 2024TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Biascited by 4
  38. 2025FLOSS: Free Lunch in Open-vocabulary Semantic Segmentationcited by 4

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Discovering Meaningful Units with Visually Grounded Semantics from Image Captions
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  3. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  4. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
  5. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
  6. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
  7. 2025Context Matters: Learning Global Semantics via Object-Centric Representation
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding