mlsanityData to 7 Oct 2026

LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model

arXiv 2312.17240 ↗First posted 28 Dec 2023cs.CVcited by 70

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

While LISA effectively bridges the gap between segmentation and large language models to enable reasoning segmentation, it poses certain limitations: unable to distinguish different instances of the target region, and constrained by the pre-defined textual response formats. In this work, we introduce LISA++, an update to the existing LISA model, focusing on improving core functionalities while keeping the base architecture intact. The main enhancements in LISA++ include: 1) Enhanced Segmentation: The instance segmentation ability has been added, providing a more detailed scene analysis along with the existing multi-region semantic segmentation. 2) More Natural Conversation: Improved capability for multi-turn dialogue, with the ability to incorporate segmentation results directly into text responses, i.e., Segmentation in Dialogue (SiD). These improvements are achieved by curating the existing samples of generic segmentation datasets, aimed specifically at enhancing the segmentation and conversational skills without structural change and additional data sources. Comparative analysis with the original LISA model shows significant advancements in these areas, positioning LISA++ as a notable upgrade in visual understanding and interaction. LISA++'s adaptability and improved features highlight the versatility of the mask-as-embedding paradigm proposed by LISA, and the potential as a foundational model for diverse applications.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 42 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

39 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Lenna: Language Enhanced Reasoning Detection Assistant5 Dec
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023DetGPT: Detect What You Need via Reasoning23 May
  4. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
22 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  12. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  13. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  14. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  16. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  17. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  18. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  19. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  20. 2021PFENet++: Boosting Few-shot Semantic Segmentation with the Noise-filtered Context-aware Prior Maskcited by 31
  21. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31
  22. 2022Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directionscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 27; 2026: 19.

70 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  2. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  4. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  5. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  6. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  7. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  8. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47
  9. 2024OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentationcited by 43
  10. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  11. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  12. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 40
38 more
  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2024Unified Language-driven Zero-shot Domain Adaptationcited by 30
  3. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  4. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  5. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  6. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  7. 2024ViLLa: Video Reasoning Segmentation with Large Language Modelcited by 23
  8. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 19
  9. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  10. 2024InsightEdit: Towards Better Instruction Following for Image Editingcited by 17
  11. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  12. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  13. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  14. 2025Large Language Model for Lossless Image Compression with Visual Promptscited by 15
  15. 2025OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Promptscited by 14
  16. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentationcited by 13
  17. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  18. 2024Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehensioncited by 10
  19. 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
  20. 2025Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twinscited by 9
  21. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  22. 2025Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoningcited by 8
  23. 2024SeafloorAI: A Large-scale Vision-Language Dataset for Seafloor Geological Surveycited by 6
  24. 2025Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensingcited by 6
  25. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
  26. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentationcited by 5
  27. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  28. 2025Joint Lossless Compression and Steganography for Medical Images via Large Language Modelscited by 5
  29. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  30. 2026Vision as Unified Multimodal Generationcited by 5
  31. 2024Dynamic Pseudo Label Optimization in Point-Supervised Nuclei Segmentationcited by 4
  32. 2025GRASP: Geospatial pixel Reasoning viA Structured Policy learningcited by 4
  33. 2025FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learningcited by 4
  34. 2026Efficient Reasoning with Balanced Thinkingcited by 4
  35. 2026WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Modelscited by 4
  36. 2025SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentationcited by 3
  37. 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Designcited by 3
  38. 2024TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editingcited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
  2. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models
  3. 2024Empowering Segmentation Ability to Multi-modal Large Language Models
  4. 2025Pixel-Level Reasoning Segmentation via Multi-turn Conversations
  5. 2026CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation
  6. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  7. 2026Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
  8. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence