mlsanityData to 7 Oct 2026

Generalized Decoding for Pixel, Image, and Language

arXiv 2212.11270 ↗First posted 21 Dec 2022cs.CV cs.CLcited by 292

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Further, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level visual-semantic understanding space, without any pseudo-labeling. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on eight datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient finetuning and novel task composition (e.g., referring captioning and image editing). Code, demo, video, and visualization are available at https://x-decoder-vl.github.io.

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Builds on 84 of its 106 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

84 of 106 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021Grounded Language-Image Pre-trainingcited by 1,147
  16. 2016Modeling Context in Referring Expressionscited by 1,054
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  19. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  20. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  21. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  22. 2018Panoptic Segmentationcited by 796
  23. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  24. 2022Language-driven Semantic Segmentationcited by 711
  25. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  27. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  28. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  29. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  30. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  31. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  32. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  33. 2021Image Segmentation Using Text and Image Promptscited by 509
  34. 2019YOLACT: Real-time Instance Segmentationcited by 441
  35. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  36. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  37. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  38. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 85; 2024: 94; 2025: 78; 2026: 34.

292 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  5. 2023Segment Everything Everywhere All at Oncecited by 546
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  7. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2023Segment Anything in High Qualitycited by 373
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  11. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  12. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
38 more
  1. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  2. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  3. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  4. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  5. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  6. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  7. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  8. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  9. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  10. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  11. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  13. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  14. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  15. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  16. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  17. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  18. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  19. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  20. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  21. 2024Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observationcited by 97
  22. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  23. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  24. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  25. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  26. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  27. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  28. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  29. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  30. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  31. 2023SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Modelcited by 61
  32. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  33. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 59
  34. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  35. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  36. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  37. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 53
  38. 2024Visual Agents as Fast and Slow Thinkerscited by 53

50 most cited of 292 All 292 (JSON)