# Revealing the Semantic Selection Gap in DINOv3 through Training-Free Few-Shot Segmentation

arXiv: https://arxiv.org/abs/2602.07550 · HTML: https://mlsanity.com/paper/2602.07550/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Semantic Segmentation of Underwater Imagery: Dataset and Benchmark](https://mlsanity.com/paper/2004.01241/) (2020)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation](https://mlsanity.com/paper/2207.08549/) (2022)
- [Feature-Proxy Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2210.06908/) (2022)
- [Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2210.06780/) (2022)
- [Bridge the Points: Graph-based Few-shot Segment Anything Semantically](https://mlsanity.com/paper/2410.06964/) (2024)
- [Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining](https://mlsanity.com/paper/2401.08407/) (2024)
- [Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2402.17614/) (2024)
- [MFNet: Multi-class Few-shot Segmentation Network with Pixel-wise Metric Learning](https://mlsanity.com/paper/2111.00232/) (2021)
- [Adaptive Layer Selection for Efficient Vision Transformer Fine-Tuning](https://mlsanity.com/paper/2408.08670/) (2024)
- [Label Anything: Multi-Class Few-Shot Semantic Segmentation with Visual Prompts](https://mlsanity.com/paper/2407.02075/) (2024)
- [SAM-MPA: Applying SAM to Few-shot Medical Image Segmentation using Mask Propagation and Auto-prompting](https://mlsanity.com/paper/2411.17363/) (2024)
- [ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads](https://mlsanity.com/paper/2506.03433/) (2025)

## Built on by (most cited first, 3 of 3)

- [SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation](https://mlsanity.com/paper/2605.17630/) (2026)
- [FROST: Training-Free Few-Shot Segmentation with Frozen Features and Nonparametric Statistics](https://mlsanity.com/paper/2606.31136/) (2026)
- [Queries Knew More Than We Thought: Uncovering Latent Knowledge in Segmentation Models](https://mlsanity.com/paper/2609.20283/) (2026)

## Abstract (arXiv)

Recent self-supervised Vision Transformers (ViTs), such as DINOv3, provide rich feature representations for dense vision tasks. This study investigates the intrinsic few-shot semantic segmentation (FSS) capabilities of frozen DINOv3 features through a training-free baseline, FSSDINO, utilizing class-specific prototypes and Gram-matrix refinement. Our results across binary, multi-class, and cross-domain (CDFSS) benchmarks demonstrate that this minimal approach, applied to the final backbone layer, is highly competitive with specialized methods involving complex decoders or test-time adaptation. Crucially, we conduct an Oracle-guided layer analysis, identifying a significant performance gap between the standard last-layer features and globally optimal intermediate representations. We reveal a "Safest vs. Optimal" dilemma: while the Oracle proves higher performance is attainable, matching the results of compute-intensive adaptation methods, current unsupervised and support-guided selection metrics consistently yield lower performance than the last-layer baseline. This characterizes a "Semantic Selection Gap" in Foundation Models, a disconnect where traditional heuristics fail to reliably identify high-fidelity features. Our work establishes the "Last-Layer" as a deceptively strong baseline and provides a rigorous diagnostic of the latent semantic potentials in DINOv3.The code is publicly available at https://github.com/hussni0997/fssdino.