mlsanityData to 7 Oct 2026

LISA: Reasoning Segmentation via Large Language Model

arXiv 2308.00692 ↗First posted 1 Aug 2023cs.CVcited by 770

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Although perception systems have made remarkable advancements in recent years, they still rely on explicit human instruction or pre-defined categories to identify the target objects before executing visual recognition tasks. Such systems cannot actively reason and comprehend implicit user intention. In this work, we propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction-mask data samples, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of multimodal Large Language Models (LLMs) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a <SEG> token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving complex reasoning and world knowledge. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation data samples results in further performance enhancement. Both quantitative and qualitative experiments show our method effectively unlocks new reasoning segmentation capabilities for multimodal LLMs. Code, models, and data are available at https://github.com/dvlab-research/LISA.

Key facts. Cited by 770 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 68 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

58 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DetGPT: Detect What You Need via Reasoning23 May
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar

Other references · 55

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Pyramid Scene Parsing Networkcited by 2,472
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  5. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  13. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  14. 2018Panoptic Segmentationcited by 796
  15. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  16. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2023Segment Everything Everywhere All at Oncecited by 546
  19. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  20. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  21. 2019Panoptic-DeepLabcited by 362
  22. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  23. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  25. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  26. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  27. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  28. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  29. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  30. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  31. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  32. 2021K-Net: Towards Unified Image Segmentationcited by 211
  33. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  34. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  35. 2015ParseNet: Looking Wider to See Bettercited by 176
  36. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  37. 2023PACO: Parts and Attributes of Common Objectscited by 146
  38. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 57; 2024: 203; 2025: 311; 2026: 199.

770 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  5. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  6. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  7. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  11. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  12. 2024OmniGen: Unified Image Generationcited by 388
38 more
  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  2. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  6. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  10. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  11. 2023Video Understanding with Large Language Models: A Surveycited by 203
  12. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  13. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  14. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  15. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  16. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  17. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  18. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  19. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  20. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  21. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  22. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  23. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  24. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  25. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  26. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  27. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  28. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  29. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  30. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  31. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  32. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  33. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  34. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  35. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  36. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  37. 2024FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Modelscited by 111
  38. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107

50 most cited of 770 All 770 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
  2. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models
  3. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  4. 2025Reasoning Segmentation for Images and Videos: A Survey
  5. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models
  6. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentation
  7. 2025NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation
  8. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation