mlsanityData to 7 Oct 2026

Visual In-Context Prompting

arXiv 2311.13601 ↗First posted 22 Nov 2023cs.CV cs.AI cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain. Existing visual prompting methods focus on referring segmentation to segment the most relevant object, falling short of addressing many generic vision tasks like open-set segmentation and detection. In this paper, we introduce a universal visual in-context prompting framework for both tasks. In particular, we build on top of an encoder-decoder architecture, and develop a versatile prompt encoder to support a variety of prompts like strokes, boxes, and points. We further enhance it to take an arbitrary number of reference image segments as the context. Our extensive explorations show that the proposed visual in-context prompting elicits extraordinary referring and generic segmentation capabilities to refer and detect, yielding competitive performance to close-set in-domain datasets and showing promising results on many open-set segmentation datasets. By joint training on COCO and SA-1B, our model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be available at https://github.com/UX-Decoder/DINOv.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

37 of 45 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
25 more
  1. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  2. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  3. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  4. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  5. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  6. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  7. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  8. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  9. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  10. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  11. 2023Personalize Segment Anything Model with One Shotcited by 250
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  13. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  14. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  15. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  16. 2023SegGPT: Segmenting Everything In Contextcited by 193
  17. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  18. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  19. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  20. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  21. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  22. 2018A Generative Appearance Model for End-to-end Video Object Segmentationcited by 66
  23. 2021DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervisioncited by 59
  24. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  25. 2022SWEM: Towards Real-Time Video Object Segmentation with Sequential Weighted Expectation-Maximizationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 19; 2026: 14.

52 papers.

Sort:

By area:

  1. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  4. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  5. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  6. 2026Ultralytics YOLO26: Unified Real-Time End-to-End Vision Modelscited by 47
  7. 2025YOLOE: Real-Time Seeing Anythingcited by 36
  8. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  9. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  10. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  11. 2024A Simple Image Segmentation Framework via In-Context Examplescited by 19
  12. 2024TAPTRv2: Attention-based Position Update Improves Tracking Any Pointcited by 18
38 more
  1. 2024CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentationcited by 14
  2. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  3. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  4. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  5. 2024TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Videocited by 8
  6. 2025Can Large Vision-Language Models Detect Images Copyright Infringement from GenAI?cited by 8
  7. 2024There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Taskscited by 7
  8. 2025Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learningcited by 7
  9. 2025Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approachcited by 6
  10. 2025Unified Open-World Segmentation with Multi-Modal Promptscited by 6
  11. 2024Towards Adversarial Robustness via Debiased High-Confidence Logit Alignmentcited by 5
  12. 2024Search and Detect: Training-Free Long Tail Object Detection via Web-Image Retrievalcited by 5
  13. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  14. 2025DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Modelscited by 5
  15. 2025Scalable In-Context Q-Learningcited by 5
  16. 2025Text-guided Visual Prompt DINO for Generic Segmentationcited by 5
  17. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  18. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentationcited by 4
  19. 2024LOCR: Location-Guided Transformer for Optical Character Recognitioncited by 3
  20. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Modelscited by 3
  21. 2025VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Modelscited by 3
  22. 2026PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Trainingcited by 3
  23. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learningcited by 2
  24. 2025Omni-Referring Image Segmentationcited by 2
  25. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 2
  26. 2026Scaling Dense Event-Stream Pretraining from Visual Foundation Modelscited by 2
  27. 2026From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coachingcited by 2
  28. 2026DETR-ViP: Detection Transformer with Robust Discriminative Visual Promptscited by 2
  29. 2025Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentationcited by 1
  30. 2025Towards Efficient Multi-Scale Deformable Attention on NPUcited by 1
  31. 2024Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variationscited by 0
  32. 2024Cycle-Consistency Uncertainty Estimation for Visual Prompting based One-Shot Defect Segmentationcited by 0
  33. 2025ZERO: Industry-ready Vision Foundation Model with Multi-modal Promptscited by 0
  34. 2026PAND: Prompt-Aware Neighborhood Distillation for Lightweight Fine-Grained Visual Classificationcited by 0
  35. 2026Transferring Physical Priors into Remote Sensing Segmentation via Large Language Modelscited by 0
  36. 2026UniSpector: Towards Universal Open-set Defect Recognition via Spectral-Contrastive Visual Promptingcited by 0
  37. 2026PrAda: Few-Shot Visual Adaptation for Text-Prompted Segmentationcited by 0
  38. 2026TAP-VLA: Tactile Annotation Prompting for Vision Language Action Modelscited by 0

50 most cited of 52 All 52 (JSON)