mlsanityData to 14 Oct 2026

Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement

arXiv 2503.06520 ↗First posted 9 Mar 2025cs.CV cs.MMcited by 239

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Traditional methods for reasoning segmentation rely on supervised fine-tuning with categorical labels and simple descriptions, limiting its out-of-domain generalization and lacking explicit reasoning processes. To address these limitations, we propose Seg-Zero, a novel framework that demonstrates remarkable generalizability and derives explicit chain-of-thought reasoning through cognitive reinforcement. Seg-Zero introduces a decoupled architecture consisting of a reasoning model and a segmentation model. The reasoning model interprets user intentions, generates explicit reasoning chains, and produces positional prompts, which are subsequently used by the segmentation model to generate precious pixel-level masks. We design a sophisticated reward mechanism that integrates both format and accuracy rewards to effectively guide optimization directions. Trained exclusively via reinforcement learning with GRPO and without explicit reasoning data, Seg-Zero achieves robust zero-shot generalization and exhibits emergent test-time reasoning capabilities. Experiments show that Seg-Zero-7B achieves a zero-shot performance of 57.5 on the ReasonSeg benchmark, surpassing the prior LISA-7B by 18%. This significant improvement highlights Seg-Zero's ability to generalize across domains while presenting an explicit reasoning process.

Key facts. Cited by 239 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 44 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

38 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos29 Sep
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
22 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2016Pyramid Scene Parsing Networkcited by 2,479
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  8. 2016Modeling Context in Referring Expressionscited by 1,063
  9. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  13. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  14. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  16. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  17. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  19. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  20. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  21. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  22. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 131; 2026: 108.

239 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  2. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  5. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  6. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  8. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  9. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  11. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  12. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
38 more
  1. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  2. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  3. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  4. 2025Detect Anything via Next Point Predictioncited by 73
  5. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  6. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 61
  7. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  8. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  9. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 52
  10. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
  11. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  12. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 43
  13. 2026Image Generators are Generalist Vision Learnerscited by 43
  14. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 41
  15. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  16. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 33
  17. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  18. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  19. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  20. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  21. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  22. 2025RemoteReasoner: Towards Unifying Geospatial Reasoning Workflowcited by 28
  23. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 27
  24. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 26
  25. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  26. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 26
  27. 2025Skyra: AI-Generated Video Detection via Grounded Artifact Reasoningcited by 25
  28. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  29. 2025Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Modelcited by 23
  30. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  31. 2025Multimodal Referring Segmentation: A Surveycited by 22
  32. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  33. 2025WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learningcited by 21
  34. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 21
  35. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  36. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 20
  37. 2025HunyuanOCR Technical Reportcited by 20
  38. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19

50 most cited of 239 All 239 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning
  2. 2026GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
  3. 2024CoReS: Orchestrating the Dance of Reasoning and Segmentation
  4. 2026SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning
  5. 2025VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning
  6. 2026Short Chains, Deep Thoughts: Balancing Reasoning Efficiency and Intra-Segment Capability via Split-Merge Optimization
  7. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  8. 2026MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Grounding