# Toward Robust In-Context Segmentation via Concept Guidance

arXiv: https://arxiv.org/abs/2606.28149 · HTML: https://mlsanity.com/paper/2606.28149/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation](https://mlsanity.com/paper/1907.12347/) (2019)
- [What Makes Good Examples for Visual In-Context Learning?](https://mlsanity.com/paper/2301.13670/) (2023)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [RobustSAM: Segment Anything Robustly on Degraded Images](https://mlsanity.com/paper/2406.09627/) (2024)
- [Bridge the Points: Graph-based Few-shot Segment Anything Semantically](https://mlsanity.com/paper/2410.06964/) (2024)
- [SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation](https://mlsanity.com/paper/2311.14671/) (2023)
- [A Simple Image Segmentation Framework via In-Context Examples](https://mlsanity.com/paper/2410.04842/) (2024)
- [Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation](https://mlsanity.com/paper/2410.02369/) (2024)
- [Show and Segment: Universal Medical Image Segmentation via In-Context Learning](https://mlsanity.com/paper/2503.19359/) (2025)
- [Towards Global Optimal Visual In-Context Learning Prompt Selection](https://mlsanity.com/paper/2405.15279/) (2024)
- [Visual Prompt Selection for In-Context Learning Segmentation](https://mlsanity.com/paper/2407.10233/) (2024)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)
- [SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation](https://mlsanity.com/paper/2505.21795/) (2025)

## Abstract (arXiv)

In-context segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters. Despite recent progress, prior ICS studies have largely overlooked a critical aspect: system robustness, ie, whether the model can produce stable segmentation results for the same query under different references. In this work, we revisit ICS from the robustness perspective and introduce a novel paradigm, Concept-Guided In-Context Segmentation (CG-ICS), which performs segmentation by extracting high-level semantic concepts from references rather than relying solely on low-level visual matching. Specifically, CG-ICS introduces a concept reasoning module that uses an MLLM to propose candidates and a SAM3-driven scoring function with tree-search refinement to select reliable textual concepts, together with a parallel visual exemplar route that provides query-side spatial grounding via a simple context construction. Both the textual concept and the visual exemplar are then used to activate the segmentation capability of a frozen SAM3 backbone. Extensive experiments on standard ICS benchmarks demonstrate that CG-ICS not only achieves state-of-the-art accuracy but also substantially improves robustness, yielding a more reliable ICS system with significantly reduced variance across diverse reference choices. Code is available at https://github.com/Kakarot1103/CG-ICS.