mlsanityData to 14 Oct 2026

FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation

arXiv 2303.17225 ↗First posted 30 Mar 2023cs.CVcited by 95

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, open-vocabulary learning has emerged to accomplish segmentation for arbitrary categories of text-based descriptions, which popularizes the segmentation system to more general-purpose application scenarios. However, existing methods devote to designing specialized architectures or parameters for specific segmentation tasks. These customized design paradigms lead to fragmentation between various segmentation tasks, thus hindering the uniformity of segmentation models. Hence in this paper, we propose FreeSeg, a generic framework to accomplish Unified, Universal and Open-Vocabulary Image Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and employs the same architecture and parameters to handle diverse segmentation tasks seamlessly in the inference procedure. Additionally, adaptive prompt learning facilitates the unified model to capture task-aware and category-sensitive concepts, improving model robustness in multi-task and varied scenarios. Extensive experimental results demonstrate that FreeSeg establishes new state-of-the-art results in performance and generalization on three segmentation tasks, which outperforms the best task-specific architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen class on COCO.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

36 of 49 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  10. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
24 more
  1. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  2. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  3. 2022Language-driven Semantic Segmentationcited by 717
  4. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  5. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  6. 2019Zero-Shot Semantic Segmentationcited by 245
  7. 2020Universal Source-Free Domain Adaptationcited by 220
  8. 2021K-Net: Towards Unified Image Segmentationcited by 212
  9. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  10. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  11. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  12. 2020Tent: Fully Test-time Adaptation by Entropy Minimizationcited by 69
  13. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  14. 2020PAMS: Quantized Super-Resolution via Parameterized Max Scalecited by 48
  15. 2020Tree-Structured Policy based Progressive Reinforcement Learning for Temporally Language Grounding in Videocited by 47
  16. 2021Zero-Shot Instance Segmentationcited by 40
  17. 2020Background Learnable Cascade for Zero-Shot Object Detectioncited by 31
  18. 2021Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentationcited by 29
  19. 2021Online Multi-Granularity Distillation for GAN Compressioncited by 24
  20. 2019OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networkscited by 18
  21. 2020Fine-Grained Image Captioning with Global-Local Discriminative Objectivecited by 12
  22. 2022TRT-ViT: TensorRT-oriented Vision Transformercited by 10
  23. 2017Design of a Very Compact CNN Classifier for Online Handwritten Chinese Character Recognition Using DropWeight and Global Poolingcited by 5
  24. 2022Multi-Granularity Distillation Scheme Towards Lightweight Semi-Supervised Semantic Segmentationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 41; 2025: 25; 2026: 9.

95 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  5. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  6. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 75
  7. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  8. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  9. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  10. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentationcited by 57
  11. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
38 more
  1. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  2. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  3. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  4. 2023Tokenize Anything via Promptingcited by 31
  5. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Modelscited by 29
  6. 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 27
  7. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 22
  8. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  9. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 20
  10. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  11. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  12. 2024Open Panoramic Segmentationcited by 17
  13. 2024Open-Vocabulary Audio-Visual Semantic Segmentationcited by 16
  14. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  15. 2023Towards Training-free Open-world Segmentation via Image Prompt Foundation Modelscited by 15
  16. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  17. 2023ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Promptingcited by 15
  18. 2025MARIS: Marine Open-Vocabulary Instance Segmentation with Geometric Enhancement and Semantic Alignmentcited by 15
  19. 2023AlignDet: Aligning Pre-training and Fine-tuning in Object Detectioncited by 14
  20. 2024CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentationcited by 14
  21. 20243D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillationcited by 13
  22. 2024UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understandingcited by 13
  23. 2024CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentationcited by 13
  24. 2024Segment Any 3D Object with Languagecited by 13
  25. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 13
  26. 2023Relevant Intrinsic Feature Enhancement Network for Few-Shot Semantic Segmentationcited by 11
  27. 2024JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environmentscited by 11
  28. 2024Open-Vocabulary Remote Sensing Image Semantic Segmentationcited by 11
  29. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  30. 2023CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentationcited by 9
  31. 2024PosSAM: Panoptic Open-vocabulary Segment Anythingcited by 9
  32. 2024OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentationcited by 9
  33. 2024Generalization Boosted Adapter for Open-Vocabulary Segmentationcited by 9
  34. 2023Towards Open-Ended Visual Recognition with Large Language Modelcited by 8
  35. 2025SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentationcited by 8
  36. 2023OV-VG: A Benchmark for Open-Vocabulary Visual Groundingcited by 7
  37. 2023OpenSD: Unified Open-Vocabulary Segmentation and Detectioncited by 7
  38. 2024Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalitiescited by 7

50 most cited of 95 All 95 (JSON)