mlsanityData to 14 Oct 2026

PixelLM: Pixel Reasoning with Large Multimodal Model

arXiv 2312.02228 ↗First posted 4 Dec 2023cs.CVcited by 222

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

While large multimodal models (LMMs) have achieved remarkable progress, generating pixel-level masks for image reasoning tasks involving multiple open-world targets remains a challenge. To bridge this gap, we introduce PixelLM, an effective and efficient LMM for pixel-level reasoning and understanding. Central to PixelLM is a novel, lightweight pixel decoder and a comprehensive segmentation codebook. The decoder efficiently produces masks from the hidden embeddings of the codebook tokens, which encode detailed target-relevant information. With this design, PixelLM harmonizes with the structure of popular LMMs and avoids the need for additional costly segmentation models. Furthermore, we propose a target refinement loss to enhance the model's ability to differentiate between multiple targets, leading to substantially improved mask quality. To advance research in this area, we construct MUSE, a high-quality multi-target reasoning segmentation benchmark. PixelLM excels across various pixel-level image reasoning and understanding tasks, outperforming well-established methods in multiple benchmarks, including MUSE, single- and multi-referring segmentation. Comprehensive ablations confirm the efficacy of each proposed component. All code, models, and datasets will be publicly available.

Key facts. Cited by 222 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 40 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

30 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2023Segment Anythingcited by 8,234
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
17 more
  1. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  4. 2023Instruction Tuning with GPT-4cited by 686
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  8. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  9. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  11. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  12. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  13. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  14. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  15. 2023PACO: Parts and Attributes of Common Objectscited by 146
  16. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  17. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 102; 2026: 74.

222 papers.

Sort:

By area:

  1. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  2. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  6. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  7. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 108
  8. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  9. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  10. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 89
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  12. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
38 more
  1. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  2. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  3. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  4. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  5. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  6. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  7. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 53
  8. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
  9. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  10. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  11. 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 44
  12. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 43
  13. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  14. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  15. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 36
  16. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  17. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  18. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 34
  19. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  20. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  21. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  22. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  23. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  24. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 29
  25. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 28
  26. 2025MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentationcited by 28
  27. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  28. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  29. 2024ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Modelscited by 27
  30. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  31. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  32. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 22
  33. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  34. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  35. 2025Multimodal Referring Segmentation: A Surveycited by 22
  36. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  37. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  38. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understandingcited by 19

50 most cited of 222 All 222 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  2. 2026Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
  3. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation
  4. 2026UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
  5. 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
  6. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentation
  7. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  8. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning