mlsanityData to 14 Oct 2026

CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models

arXiv 2412.19331 ↗First posted 26 Dec 2024cs.CV cs.AI cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advances in Large Vision-Language Models (LVLMs) have enabled general-purpose vision tasks through visual instruction tuning. While existing LVLMs can generate segmentation masks from text prompts for single images, they struggle with segmentation-grounded reasoning across images, especially at finer granularities such as object parts. In this paper, we introduce the new task of part-focused semantic co-segmentation, which involves identifying and segmenting common objects, as well as common and unique object parts across images. To address this task, we present CALICO, the first LVLM designed for multi-image part-level reasoning segmentation. CALICO features two key components, a novel Correspondence Extraction Module that identifies semantic part-level correspondences, and Correspondence Adaptation Modules that embed this information into the LVLM to facilitate multi-image understanding in a parameter-efficient manner. To support training and evaluation, we curate MixedParts, a large-scale multi-image segmentation dataset containing ~2.4M samples across ~44K images spanning diverse object and part categories. Experimental results demonstrate that CALICO, with just 0.3% of its parameters finetuned, achieves strong performance on this challenging task.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 80 references that are in the map. First posted on arXiv 26 Dec 2024.

Builds on

63 of 80 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model19 Dec
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  5. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2017Mask R-CNNcited by 7,277
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2023Vision Transformers Need Registerscited by 871
  7. 2018Panoptic Segmentationcited by 802
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  9. 2023Segment Everything Everywhere All at Oncecited by 548
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  11. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  12. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  13. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  14. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  15. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  16. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  17. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  18. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  19. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  20. 2023PACO: Parts and Attributes of Common Objectscited by 146
  21. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  22. 2019SCOPS: Self-Supervised Co-Part Segmentationcited by 100
  23. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 77
  24. 2018Deep Feature Factorization For Concept Discoverycited by 64
  25. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 60
  26. 2021Unsupervised Part Discovery from Contrastive Reconstructioncited by 57
  27. 2021Part-aware Panoptic Segmentationcited by 56
  28. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  29. 2018Parsing R-CNN for Instance-Level Human Analysiscited by 53
  30. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  31. 2022Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentationcited by 38
  32. 2020Renovating Parsing R-CNN for Accurate Multiple Human Parsingcited by 37
  33. 2023Vocabulary-free Image Classificationcited by 35
  34. 2021Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsingcited by 31
  35. 2023OV-PARTS: Towards Open-Vocabulary Part Segmentationcited by 31
  36. 2019Learning Semantic Neural Tree for Human Parsingcited by 29
  37. 2021Unsupervised Part Segmentation through Disentangling Appearance and Shapecited by 29
  38. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoisingcited by 2
  2. 2025LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentationcited by 1
  3. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentationcited by 1
  4. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
  2. 2026SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
  3. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentation
  4. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model
  5. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation
  6. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models
  7. 2025Reasoning Segmentation for Images and Videos: A Survey
  8. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation