mlsanityData to 7 Oct 2026

Side Adapter Network for Open-Vocabulary Semantic Segmentation

arXiv 2302.12242 ↗First posted 23 Feb 2023cs.CV cs.AIcited by 296

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation. The code will be available at https://github.com/MendelXu/SAN.

Key facts. Cited by 296 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

36 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  2. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
20 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2022Language-driven Semantic Segmentationcited by 711
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  13. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  14. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  15. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  16. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  17. 2019Zero-Shot Semantic Segmentationcited by 245
  18. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  19. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  20. 2017Open Vocabulary Scene Parsingcited by 76

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 51; 2024: 112; 2025: 82; 2026: 51.

296 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  4. 2023Segment Everything Everywhere All at Oncecited by 546
  5. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  6. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  7. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  8. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  9. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  10. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  11. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
38 more
  1. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  2. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  3. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  4. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  5. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  6. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  7. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  8. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  9. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  10. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  11. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  12. 2024CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognitioncited by 61
  13. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  14. 2024PF3plat: Pose-Free Feed-Forward 3D Gaussian Splattingcited by 60
  15. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  16. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  17. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  18. 2023Open-Vocabulary Segmentation with Semantic-Assisted Calibrationcited by 49
  19. 2024Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognitioncited by 49
  20. 2023A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detectioncited by 47
  21. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  22. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  23. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  24. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
  25. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  26. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  27. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  28. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  29. 2024KVQ: Kwai Video Quality Assessment for Short-form Videoscited by 38
  30. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  31. 2024Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Modelcited by 37
  32. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  33. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  34. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  35. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  36. 2024Open-Vocabulary Semantic Segmentation with Image Embedding Balancingcited by 35
  37. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  38. 2025TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigationcited by 35

50 most cited of 296 All 296 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  2. 2025Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation
  3. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  4. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
  5. 2025A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIP
  6. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
  7. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements
  8. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation