mlsanityData to 14 Oct 2026

Open-Vocabulary Panoptic Segmentation Using BERT Pre-Training of Vision-Language Multiway Transformer Model

arXiv 2412.18917 ↗First posted 25 Dec 2024cs.CV cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Open-vocabulary panoptic segmentation remains a challenging problem. One of the biggest difficulties lies in training models to generalize to an unlimited number of classes using limited categorized training data. Recent popular methods involve large-scale vision-language pre-trained foundation models, such as CLIP. In this paper, we propose OMTSeg for open-vocabulary segmentation using another large-scale vision-language pre-trained model called BEiT-3 and leveraging the cross-modal attention between visual and linguistic features in BEiT-3 to achieve better performance. Experiments result demonstrates that OMTSeg performs favorably against state-of-the-art models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 12 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

12 of 12 references.

References · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2022Language-driven Semantic Segmentationcited by 717
  7. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  9. 2019Zero-Shot Semantic Segmentationcited by 245
  10. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  11. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  12. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LiPS: Lightweight Panoptic Segmentation for Resource-Constrained Roboticscited by 0
  2. 2026EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Passcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  2. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
  3. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  4. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  5. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation
  6. 2025OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoning
  7. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  8. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation