mlsanityData to 7 Oct 2026

SegLLM: Multi-round Reasoning Segmentation

arXiv 2410.18923 ↗First posted 24 Oct 2024cs.CV cs.AIcited by 25

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation to previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLLM to respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi-round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-round reasoning segmentation data enhances performance on standard single-round referring segmentation and localization tasks, resulting in a 5.5% increase in cIoU for referring expression segmentation and a 4.5% improvement in [email protected] for referring expression localization.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 54 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

46 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  2. 2023DetGPT: Detect What You Need via Reasoning23 May
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2017Mask R-CNNcited by 7,259
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
31 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  15. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  16. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2023Segment Everything Everywhere All at Oncecited by 546
  19. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  20. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  21. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  22. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  23. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  24. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  25. 2023PACO: Parts and Attributes of Common Objectscited by 146
  26. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  27. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  28. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  29. 2021Part-aware Panoptic Segmentationcited by 56
  30. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  31. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 11.

25 papers.

Sort:

By area:

  1. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
  2. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  3. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  4. 2025Multimodal Referring Segmentation: A Surveycited by 22
  5. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  6. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  7. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  8. 2025PRS-Med: Position Reasoning Segmentation in Medical Imagingcited by 8
  9. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  10. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  11. 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 5
  12. 2026Discriminative Perception via Anchored Description for Reasoning Segmentationcited by 3
13 more
  1. 2026Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Modelscited by 2
  2. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  3. 2025MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Imagescited by 1
  4. 2025CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentationcited by 1
  5. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  6. 2026B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentationcited by 1
  7. 2026An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentationcited by 1
  8. 2025Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learningcited by 0
  9. 2025Fast Reasoning Segmentation for Images and Videoscited by 0
  10. 2025Chunking Strategies for Multimodal AI Systemscited by 0
  11. 2026MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Trainingcited by 0
  12. 2026COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decompositioncited by 0
  13. 20260.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models
  2. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  3. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning
  4. 2026CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation
  5. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
  6. 2026SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation
  7. 2026CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
  8. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning