# Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting

arXiv: https://arxiv.org/abs/2512.24702 · HTML: https://mlsanity.com/paper/2512.24702/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 34 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 30 of 32)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [GSVA: Generalized Segmentation via Multimodal Large Language Models](https://mlsanity.com/paper/2312.10103/) (2023)
- [PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model](https://mlsanity.com/paper/2403.14598/) (2024)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor](https://mlsanity.com/paper/2312.07661/) (2023)
- [SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation](https://mlsanity.com/paper/2409.10542/) (2024)
- [EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model](https://mlsanity.com/paper/2406.20076/) (2024)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning](https://mlsanity.com/paper/2404.08767/) (2024)
- [Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models](https://mlsanity.com/paper/2308.16777/) (2023)
- [HyperSeg: Towards Universal Visual Segmentation with Large Language Model](https://mlsanity.com/paper/2411.17606/) (2024)
- [SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories](https://mlsanity.com/paper/2503.08625/) (2025)
- [X-SAM: From Segment Anything to Any Segmentation](https://mlsanity.com/paper/2508.04655/) (2025)
- [Universal Segmentation at Arbitrary Granularity with Language Instruction](https://mlsanity.com/paper/2312.01623/) (2023)
- [Learning Visual Grounding from Generative Vision and Language Model](https://mlsanity.com/paper/2407.14563/) (2024)
- [IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis](https://mlsanity.com/paper/2503.00936/) (2025)
- [Text Augmented Spatial-aware Zero-shot Referring Image Segmentation](https://mlsanity.com/paper/2310.18049/) (2023)
- [RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2506.04277/) (2025)
- [LENS: Learning to Segment Anything with Unified Reinforced Reasoning](https://mlsanity.com/paper/2508.14153/) (2025)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation](https://mlsanity.com/paper/2504.00356/) (2025)

## Built on by (most cited first, 4 of 4)

- [Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation](https://mlsanity.com/paper/2604.07916/) (2026)
- [Decoding the Delta: Unifying Remote Sensing Change Detection and Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2604.14044/) (2026)
- [Vision Harnessing Agent for Open Ad-hoc Segmentation](https://mlsanity.com/paper/2605.19410/) (2026)
- [CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation](https://mlsanity.com/paper/2606.03564/) (2026)

## Abstract (arXiv)

Reasoning Segmentation requires models to interpret complex, context-dependent linguistic queries to achieve pixel-level localization. Current dominant approaches rely heavily on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, SFT suffers from catastrophic forgetting and domain dependency, while RL is often hindered by training instability and rigid reliance on predefined reward functions. Although recent training-free methods circumvent these training burdens, they are fundamentally limited by a static inference paradigm. These methods typically rely on a single-pass "generate-then-segment" chain, which suffers from insufficient reasoning depth and lacks the capability to self-correct linguistic hallucinations or spatial misinterpretations. In this paper, we challenge these limitations and propose EVOL-SAM3, a novel zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process. Instead of relying on a fixed prompt, EVOL-SAM3 maintains a population of prompt hypotheses and iteratively refines them through a "Generate-Evaluate-Evolve" loop. We introduce a Visual Arena to assess prompt fitness via reference-free pairwise tournaments, and a Semantic Mutation operator to inject diversity and correct semantic errors. Furthermore, a Heterogeneous Arena module integrates geometric priors with semantic reasoning to ensure robust final selection. Extensive experiments demonstrate that EVOL-SAM3 not only substantially outperforms static baselines but also significantly surpasses fully supervised state-of-the-art methods on the challenging ReasonSeg benchmark in a zero-shot setting. The code is available at https://github.com/AHideoKuzeA/Evol-SAM3.