mlsanityData to 14 Oct 2026

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

arXiv 2403.14598 ↗First posted 21 Mar 2024cs.CVcited by 101

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. To overcome the limitation of the LMM being limited to textual output, PSALM incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. This schema includes images, task instructions, conditional prompts, and mask tokens, which enable the model to generate and classify segmentation masks effectively. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization. PSALM achieves superior results on several benchmarks, such as RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive, and further exhibits zero-shot capabilities on unseen tasks, such as open-vocabulary segmentation, generalized referring expression segmentation and video object segmentation, making a significant step towards a GPT moment in computer vision. Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing. Code and models are available at https://github.com/zamling/PSALM.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 66 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

59 of 66 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  6. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2023Segment Anythingcited by 8,234
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
38 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2016Modeling Context in Referring Expressionscited by 1,063
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  11. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  13. 2023Segment Everything Everywhere All at Oncecited by 548
  14. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  15. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  16. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  17. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  18. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  19. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  20. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  21. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  22. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  23. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  24. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  25. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  26. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  27. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  28. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  29. 2023SegGPT: Segmenting Everything In Contextcited by 194
  30. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  31. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  32. 2023PACO: Parts and Attributes of Common Objectscited by 146
  33. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  34. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 75
  35. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  36. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  37. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  38. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Modelcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 47; 2026: 39.

101 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  4. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  5. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  6. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  7. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  8. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 53
  9. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 49
  10. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 40
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  12. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 36
38 more
  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  3. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  4. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  5. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  6. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 29
  7. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  8. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  9. 2025Multimodal Referring Segmentation: A Surveycited by 22
  10. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  11. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  12. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understandingcited by 18
  13. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 15
  14. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  15. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 14
  16. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  17. 2025NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoningcited by 12
  18. 2025GroundingSuite: Measuring Complex Multi-Granular Pixel Groundingcited by 11
  19. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  20. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  21. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  22. 2024Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Modelcited by 8
  23. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8
  24. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  25. 2025Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decodercited by 8
  26. 2025O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Viewscited by 7
  27. 2025Unified Open-World Segmentation with Multi-Modal Promptscited by 7
  28. 2026Vision as Unified Multimodal Generationcited by 7
  29. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 6
  30. 2026GREx: Generalized Referring Expression Segmentation, Comprehension, and Generationcited by 6
  31. 2026DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Modelscited by 6
  32. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  33. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  34. 2025Refer to Any Segmentation Mask Group With Vision-Language Promptscited by 5
  35. 2025Cross-View Multi-Modal Segmentation @ Ego-Exo4D Challenges 2025cited by 5
  36. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  37. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 5
  38. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
  2. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
  3. 2025BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts
  4. 2024TG-LMM: Enhancing Medical Image Segmentation Accuracy through Text-Guided Large Multi-Modal Model
  5. 2024Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
  7. 2025LawDIS: Language-Window-based Controllable Dichotomous Image Segmentation
  8. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor