mlsanityData to 14 Oct 2026

Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation

arXiv 2509.01275 ↗First posted 1 Sep 2025cs.CVcited by 2

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Open-vocabulary semantic segmentation (OVSS) conducts pixel-level classification via text-driven alignment, where the domain discrepancy between base category training and open-vocabulary inference poses challenges in discriminative modeling of latent unseen category. To address this challenge, existing vision-language model (VLM)-based approaches demonstrate commendable performance through pre-trained multi-modal representations. However, the fundamental mechanisms of latent semantic comprehension remain underexplored, making the bottleneck for OVSS. In this work, we initiate a probing experiment to explore distribution patterns and dynamics of latent semantics in VLMs under inductive learning paradigms. Building on these insights, we propose X-Agent, an innovative OVSS framework employing latent semantic-aware ``agent'' to orchestrate cross-modal attention mechanisms, simultaneously optimizing latent semantic dynamic and amplifying its perceptibility. Extensive benchmark evaluations demonstrate that X-Agent achieves state-of-the-art performance while effectively enhancing the latent semantic saliency.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 38 of its 53 references that are in the map. First posted on arXiv 1 Sep 2025.

Builds on

38 of 53 references.

References · 38

  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  3. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  5. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  6. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  7. 2021Extract Free Dense Labels from CLIPcited by 583
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  11. 2019Zero-Shot Semantic Segmentationcited by 245
  12. 2019Learning to Discover Novel Visual Categories via Deep Transfer Clusteringcited by 225
26 more
  1. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  2. 2021A Unified Objective for Novel Class Discoverycited by 162
  3. 2020Automatically Discovering and Learning New Visual Categories with Ranking Statisticscited by 143
  4. 2017Learning to cluster in order to transfer across domains and taskscited by 128
  5. 2021AutoNovel: Automatically Discovering and Learning Novel Visual Categoriescited by 116
  6. 2020Learning Open Set Network with Discriminative Reciprocal Pointscited by 115
  7. 2019Multi-class Classification without Multi-class Labelscited by 100
  8. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97
  9. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  10. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  11. 2020OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in An Open Worldcited by 93
  12. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  13. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  14. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  15. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  16. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentationcited by 57
  17. 2021Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Datacited by 55
  18. 2020Few-Shot Open-Set Recognition using Meta-Learningcited by 50
  19. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  20. 2017Sparse Representation-based Open Set Recognitioncited by 36
  21. 2019P-ODN: Prototype based Open Deep Network for Open Set Recognitioncited by 28
  22. 2023Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentationcited by 28
  23. 2020Open-set Adversarial Defensecited by 27
  24. 2018Metric Learning for Novelty and Anomaly Detectioncited by 21
  25. 2023Probabilistic Prompt Learning for Dense Predictioncited by 17
  26. 2021Conterfactual Generative Zero-Shot Semantic Segmentationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentationcited by 0
  2. 2026EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Passcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  2. 2025Guideline-Consistent Segmentation via Multi-Agent Refinement
  3. 2025OVOD-Agent: A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection
  4. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation
  5. 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaboration
  6. 2023Compositional Semantics for Open Vocabulary Spatio-semantic Representations
  7. 2026dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
  8. 2026Multi-Agent Cooperative Learning for Robust Vision-Language Alignment under OOD Concepts