mlsanityData to 14 Oct 2026

Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning

arXiv 2506.22624 ↗First posted 27 Jun 2025cs.CVcited by 44

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

We present Seg-R1, a preliminary exploration of using reinforcement learning (RL) to enhance the pixel-level understanding and reasoning capabilities of large multimodal models (LMMs). Starting with foreground segmentation tasks, specifically camouflaged object detection (COD) and salient object detection (SOD), our approach enables the LMM to generate point and bounding box prompts in the next-token fashion, which are then used to guide SAM2 in producing segmentation masks. We introduce Group Relative Policy Optimization (GRPO) into the segmentation domain, equipping the LMM with pixel-level comprehension through a carefully designed training strategy. Notably, Seg-R1 achieves remarkable performance with purely RL-based training, achieving .873 S-measure on COD10K without complex model modification. Moreover, we found that pure RL training demonstrates strong open-world generalization. Despite being trained solely on foreground segmentation image-mask pairs without text supervision, Seg-R1 achieves impressive zero-shot performance on referring segmentation and reasoning segmentation tasks, with 71.4 cIoU on RefCOCOg test and 56.7 gIoU on ReasonSeg test, outperforming models fully supervised on these datasets.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 75 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

59 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding27 Jun
  3. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  4. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 54

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Segment Anythingcited by 8,234
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  13. 2016A Diagram Is Worth A Dozen Imagescited by 804
  14. 2018Panoptic Segmentationcited by 802
  15. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  16. 2023Segment Everything Everywhere All at Oncecited by 548
  17. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  18. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  19. 2018Enhanced-alignment Measure for Binary Foreground Map Evaluationcited by 370
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  21. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  22. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  23. 2015Visual Saliency Based on Multiscale Deep Featurescited by 185
  24. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  25. 2021Camouflaged Object Segmentation with Distraction Miningcited by 151
  26. 2022Zoom In and Out: A Mixed-scale Triplet Network for Camouflaged Object Detectioncited by 145
  27. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 114
  28. 2024Bilateral Reference for High-Resolution Dichotomous Image Segmentationcited by 112
  29. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  30. 2023Feature Shrinkage Pyramid for Camouflaged Object Detection with Transformerscited by 87
  31. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 86
  32. 2023ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detectioncited by 48
  33. 2023Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objectscited by 40
  34. 2022Texture-guided Saliency Distilling for Unsupervised Salient Object Detectioncited by 24
  35. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  36. 2022A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labelscited by 20
  37. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  38. 2023Explicit Visual Prompting for Universal Foreground Segmentationscited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 33.

44 papers.

Sort:

By area:

  1. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  2. 2025Multimodal Referring Segmentation: A Surveycited by 22
  3. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  4. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segmentscited by 15
  5. 2026SAMTok: Representing Any Mask with Two Wordscited by 11
  6. 2026PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learningcited by 10
  7. 2025CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentationcited by 9
  8. 2026CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoningcited by 9
  9. 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 8
  10. 2025RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocationscited by 6
  11. 2026GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentationcited by 6
  12. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
32 more
  1. 2026WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Modelscited by 5
  2. 2026TikArt: Stabilizing Aperture-Guided Fine-Grained Visual Reasoning with Reinforcement Learningcited by 4
  3. 2026SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentationcited by 4
  4. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 4
  5. 2026Urban Socio-Semantic Segmentation with Vision-Language Reasoningcited by 3
  6. 2026UniHM: Unified Dexterous Hand Manipulation with Vision Language Modelcited by 3
  7. 2026Discriminative Perception via Anchored Description for Reasoning Segmentationcited by 3
  8. 2026From Web to Pixels: Bringing Agentic Search into Visual Perceptioncited by 3
  9. 2025Grounding Everything in Tokens for Multimodal Large Language Modelscited by 2
  10. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  11. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoningcited by 2
  12. 2026Evidence-Backed Video Question Answeringcited by 2
  13. 2025Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learningcited by 1
  14. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  15. 2026Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perceptioncited by 1
  16. 2026Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentationcited by 1
  17. 2025PreferThinker: Reasoning-based Personalized Image Preference Assessmentcited by 0
  18. 2025DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detectioncited by 0
  19. 2025FishDetector-R1: Unified MLLM-Based Framework with Reinforcement Fine-Tuning for Weakly Supervised Fish Detection, Segmentation, and Countingcited by 0
  20. 2026Perception-Aware Multimodal Spatial Reasoning from Monocular Imagescited by 0
  21. 2026PhySe-RPO: Physics and Semantics Guided Relative Policy Optimization for Diffusion-Based Surgical Smoke Removalcited by 0
  22. 2026ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learningcited by 0
  23. 2026CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detectioncited by 0
  24. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  25. 2026InstanceControl: Controllable Complex Image Generation without Instance Labelingcited by 0
  26. 2026DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentationcited by 0
  27. 2026Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learningcited by 0
  28. 2026FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentationcited by 0
  29. 2026Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Modelscited by 0
  30. 2026EviRover: Reinforcing Agentic Perception Beyond a Glancecited by 0
  31. 2026ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentationcited by 0
  32. 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
  2. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  3. 2025LENS: Learning to Segment Anything with Unified Reinforced Reasoning
  4. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
  5. 2026B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation
  6. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoning
  7. 2025ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning
  8. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation