mlsanityData to 14 Oct 2026

Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models

arXiv 2607.00357 ↗First posted 1 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Personalized object localization (POL) localizes an object instance in a query image based on a few reference images with bounding-box annotations and a target object label. The pioneering method, IPLoc, solves this task through in-context inference with vision-language models (VLMs). However, it assumes that the query image always contains the target object. This assumption severely limits its applicability to real-world scenarios with many irrelevant images. To address this issue, we formulate a new task, personalized object identification and localization (POIL), by positioning POL within the broader few-shot object detection framework. POIL aims to localize the target object instance while rejecting query images that do not contain the reference object instance. We also present POIL datasets constructed from public sources. We further propose an in-context algorithm named IPLoc-ID for solving POIL with VLMs. IPLoc-ID first predicts a candidate bounding box and then determines whether it corresponds to the reference object instance. We introduce a self-posed query to connect these two steps within a single autoregressive generation framework. Through ablation studies and comprehensive experiments, we show that IPLoc-ID substantially suppresses false-positive detections on negative query images while maintaining localization performance comparable to IPLoc. Overall, IPLoc-ID effectively addresses the practical instance-level POIL task, which cannot be sufficiently solved by conventional object detection, few-shot object detection, or the localization-only IPLoc method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 49 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

43 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Teaching VLMs to Localize Specific Objects from In-context Examples20 Nov
  2. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant17 Mar
  3. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  4. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models9 Oct
  5. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models2 Jun
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2025Qwen3-VL Technical Reportcited by 2,793
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
25 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  3. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  4. 2021MetaICL: Learning to Learn In Contextcited by 454
  5. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  6. 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 425
  7. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  8. 2020Frustratingly Simple Few-Shot Object Detectioncited by 282
  9. 2019Meta R-CNN : Towards General Solver for Instance-level Few-shot Learningcited by 182
  10. 2021FSCE: Few-Shot Object Detection via Contrastive Proposal Encodingcited by 168
  11. 2021DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detectioncited by 106
  12. 2022Few-Shot Object Detection with Fully Cross-Transformercited by 64
  13. 2021Few-Shot Object Detection: A Comprehensive Surveycited by 29
  14. 2023The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Modelscited by 22
  15. 2023Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Propertiescited by 21
  16. 2024Where's Waldo: Diffusion Features for Personalized Segmentation and Retrievalcited by 18
  17. 2025Instance-Level Composed Image Retrievalcited by 12
  18. 2024Few-Shot Object Detection: Research Advances and Challengescited by 11
  19. 2023Towards More Unified In-context Visual Understandingcited by 10
  20. 2025No time to train! Training-Free Reference-Based Instance Segmentationcited by 9
  21. 2025Referring Expression Instance Retrieval and A Strong End-to-End Baselinecited by 6
  22. 2026Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Groundingcited by 5
  23. 2024Few-shot Object Localizationcited by 2
  24. 2026FSOD-VFM: Few-Shot Object Detection with Vision Foundation Models and Graph Diffusioncited by 2
  25. 2026A Closer Look at Cross-Domain Few-Shot Object Detection: Fine-Tuning Matters and Parallel Decoder Helpscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
  2. 2023Unsupervised Open-Vocabulary Object Localization in Videos
  3. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models
  4. 2024CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps
  5. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
  6. 2025VLM-Guided Visual Place Recognition for Planet-Scale Geo-Localization
  7. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Models
  8. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review