mlsanityData to 14 Oct 2026

LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning

arXiv 2404.08767 ↗First posted 12 Apr 2024cs.CV cs.LGcited by 52

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Understanding human instructions to identify the target objects is vital for perception systems. In recent years, the advancements of Large Language Models (LLMs) have introduced new possibilities for image segmentation. In this work, we delve into reasoning segmentation, a novel task that enables segmentation system to reason and interpret implicit user intention via large language model reasoning and then segment the corresponding target. Our work on reasoning segmentation contributes on both the methodological design and dataset labeling. For the model, we propose a new framework named LLM-Seg. LLM-Seg effectively connects the current foundational Segmentation Anything Model and the LLM by mask proposals selection. For the dataset, we propose an automatic data generation pipeline and construct a new reasoning segmentation dataset named LLM-Seg40K. Experiments demonstrate that our LLM-Seg exhibits competitive performance compared with existing methods. Furthermore, our proposed pipeline can efficiently produce high-quality reasoning segmentation datasets. The LLM-Seg40K dataset, developed through this pipeline, serves as a new benchmark for training and evaluating various reasoning segmentation approaches. Our code, models and dataset are at https://github.com/wangjunchi/LLMSeg.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

33 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
20 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  10. 2023The Falcon Series of Open Language Modelscited by 535
  11. 2021Image Segmentation Using Text and Image Promptscited by 512
  12. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  13. 2023Segment Anything in High Qualitycited by 376
  14. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  15. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  16. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 314
  17. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  19. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  20. 2023EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understandingcited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 34; 2026: 12.

52 papers.

Sort:

By area:

  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  2. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  4. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  5. 2025Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeekcited by 24
  6. 2025Multimodal Referring Segmentation: A Surveycited by 22
  7. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  8. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  9. 2025Online Reasoning Video Segmentation with Just-in-Time Digital Twinscited by 11
  10. 2025Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twinscited by 9
  11. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  12. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
38 more
  1. 2025MediSee: Reasoning-based Pixel-level Perception in Medical Imagescited by 8
  2. 2025PRS-Med: Position Reasoning Segmentation in Medical Imagingcited by 8
  3. 2025LangGas: Introducing Language in Selective Zero-Shot Background Subtraction for Semi-Transparent Gas Leak Detection with a New Datasetcited by 6
  4. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 5
  5. 2024ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Modelcited by 4
  6. 2025PARTONOMY: Large Multimodal Models with Part-Level Visual Understandingcited by 4
  7. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  8. 2025Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURAcited by 3
  9. 2025LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Modelscited by 3
  10. 2025CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objectscited by 3
  11. 2024TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiencycited by 2
  12. 2024Navigation with VLM framework: Towards Going to Any Languagecited by 2
  13. 2025RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregationcited by 2
  14. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  15. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  16. 2025HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentationcited by 2
  17. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  18. 2025UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretationcited by 2
  19. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2
  20. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  21. 2025SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Spacecited by 1
  22. 2025MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Imagescited by 1
  23. 2025CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentationcited by 1
  24. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Modelscited by 1
  25. 2026Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentationcited by 1
  26. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  27. 2026CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentationcited by 1
  28. 2025Decoupling the Image Perception and Multimodal Reasoning for Reasoning Segmentation with Digital Twin Representationscited by 0
  29. 2025Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentationcited by 0
  30. 2025RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiologycited by 0
  31. 2025Fast Reasoning Segmentation for Images and Videoscited by 0
  32. 2025Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shotcited by 0
  33. 2025NEURO-GUARD: Neuro-Symbolic Generalization and Unbiased Adaptive Routing for Diagnostics -- Explainable Medical AIcited by 0
  34. 2025Multi-Grained Text-Guided Image Fusion for Multi-Exposure and Multi-Focus Scenarioscited by 0
  35. 2026Revisiting Salient Object Detection from an Observer-Centric Perspectivecited by 0
  36. 2026Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMscited by 0
  37. 2026Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptationcited by 0
  38. 2026Hybrid LLM-based Intelligent Framework for Robot Task Schedulingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  2. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
  3. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models
  4. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
  5. 2024SegLLM: Multi-round Reasoning Segmentation
  6. 2026CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation
  7. 2025NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation
  8. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning