mlsanityData to 7 Oct 2026

LaSagnA: Language-based Segmentation Assistant for Complex Queries

arXiv 2404.08506 ↗First posted 12 Apr 2024cs.CVcited by 55

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements have empowered Large Language Models for Vision (vLLMs) to generate detailed perceptual outcomes, including bounding boxes and masks. Nonetheless, there are two constraints that restrict the further application of these vLLMs: the incapability of handling multiple targets per query and the failure to identify the absence of query objects in the image. In this study, we acknowledge that the main cause of these problems is the insufficient complexity of training queries. Consequently, we define the general sequence format for complex queries. Then we incorporate a semantic segmentation task in the current pipeline to fulfill the requirements of training data. Furthermore, we present three novel strategies to effectively handle the challenges arising from the direct integration of the proposed format. The effectiveness of our model in processing complex queries is validated by the comparable results with conventional methods on both close-set and open-set semantic segmentation datasets. Additionally, we outperform a series of vLLMs in reasoning and referring segmentation, showcasing our model's remarkable capabilities. We release the code at https://github.com/congvvc/LaSagnA.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 52 of its 58 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

52 of 58 references.

References · 52

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
38 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  12. 2022Language-driven Semantic Segmentationcited by 711
  13. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  14. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  15. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  18. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  19. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  20. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  21. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  22. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  23. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  24. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  25. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  26. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  27. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  28. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  29. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  30. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  31. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  32. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  33. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  34. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  35. 2021Semi-supervised Semantic Segmentation with Directional Context-aware Consistencycited by 112
  36. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  37. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  38. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 24; 2026: 19.

55 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  4. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  5. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  6. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  7. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  8. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  9. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  10. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  11. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  12. 2025Multimodal Referring Segmentation: A Surveycited by 22
38 more
  1. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  2. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  3. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  4. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  5. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  6. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  7. 2025CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentationcited by 9
  8. 2024Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Modelcited by 8
  9. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  10. 2025Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decodercited by 7
  11. 2025Latent Expression Generation for Referring Image Segmentation and Groundingcited by 6
  12. 2025A large-scale image-text dataset benchmark for farmland segmentationcited by 5
  13. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  14. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  15. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  16. 2024CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Modelscited by 4
  17. 2025LlamaSeg: Image Segmentation via Autoregressive Mask Generationcited by 4
  18. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  19. 2026SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentationcited by 4
  20. 2025Advancing Visual Large Language Model for Multi-granular Versatile Perceptioncited by 3
  21. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidencecited by 3
  22. 2024A Simple Baseline with Single-encoder for Referring Image Segmentationcited by 2
  23. 2025Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Groundercited by 2
  24. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  25. 2026Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generationcited by 2
  26. 2026Rethinking MLLM Itself as a Segmenter with a Single Segmentation Tokencited by 2
  27. 2024TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiencycited by 1
  28. 2025EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Modelscited by 1
  29. 2026GeoSeg: Training-Free Reasoning-Driven Segmentation in Remote Sensing Imagerycited by 1
  30. 2026IQA-Spider: Unifying Multi-Granularity Image Quality Assessment with Reasoning, Grounding and Referringcited by 1
  31. 2025Decoupling the Image Perception and Multimodal Reasoning for Reasoning Segmentation with Digital Twin Representationscited by 0
  32. 2025Logo-VGR: Visual Grounded Reasoning for Open-world Logo Recognitioncited by 0
  33. 2026ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentationcited by 0
  34. 2026FarmMind: Reasoning-Query-Driven Dynamic Segmentation for Farmland Remote Sensing Imagescited by 0
  35. 2026GTPBD-MM: A Global Terraced Parcel and Boundary Dataset with Multi-Modalitycited by 0
  36. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Modelscited by 0
  37. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  38. 2026Enhancing Part-Level Point Grounding for Any Open-Source MLLMscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models
  2. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  3. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
  4. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  5. 2025Refer to Any Segmentation Mask Group With Vision-Language Prompts
  6. 2026Toward Semantic-Agnostic and Shape-Aware Vision-Language Segmentation Models
  7. 2025Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughts
  8. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models