Personalized Vision via Visual In-Context Learning
Not on the map. Why?
Abstract (arXiv)
Modern vision models, trained on large-scale annotated datasets, excel at predefined tasks but struggle with personalized vision -- tasks defined at test time by users with customized objects or novel objectives. Existing personalization approaches rely on costly fine-tuning or synthetic data pipelines, which are inflexible and restricted to fixed task formats. Visual in-context learning (ICL) offers a promising alternative, yet prior methods confine to narrow, in-domain tasks and fail to generalize to open-ended personalization. We introduce Personalized In-Context Operator (PICO), a simple four-panel framework that repurposes diffusion transformers as visual in-context learners. Given a single annotated exemplar, PICO infers the underlying transformation and applies it to new inputs without retraining. To enable this, we construct VisRel, a compact yet diverse tuning dataset, showing that task diversity, rather than scale, drives robust generalization. We further propose an attention-guided seed scorer that improves reliability via efficient inference scaling. Extensive experiments demonstrate that PICO (i) surpasses fine-tuning and synthetic-data baselines, (ii) flexibly adapts to novel user-defined tasks, and (iii) generalizes across both recognition and generation.
Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
53 of 60 references.
References · 53
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
38 more
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
- 2023Adversarial Diffusion Distillationcited by 759
- 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 723
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
- 2024OmniGen: Unified Image Generationcited by 394
- 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
- 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
- 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
- 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
- 2020Multi-Scale Progressive Fusion Network for Single Image Derainingcited by 260
- 2023DiffIR: Efficient Diffusion Model for Image Restorationcited by 255
- 2022Visual Prompting via Image Inpaintingcited by 250
- 2023Personalize Segment Anything Model with One Shotcited by 250
- 2023SinSR: Diffusion-Based Image Super-Resolution in a Single Stepcited by 238
- 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
- 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 196
- 2023SegGPT: Segmenting Everything In Contextcited by 194
- 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 166
- 2024In-Context LoRA for Diffusion Transformerscited by 158
- 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
- 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
- 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
- 2022Highly Accurate Dichotomous Image Segmentationcited by 110
- 2019Dense Haze: A benchmark for image dehazing with dense-haze and haze-free imagescited by 107
- 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
- 2024Diffusion Models in Low-Level Vision: A Surveycited by 74
- 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
- 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
- 2023ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objectscited by 42
- 2025OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Datacited by 32
- 2024Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Modelcited by 30
- 2023Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matchingcited by 29
- 2025PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Datacited by 24
- 2024Where's Waldo: Diffusion Features for Personalized Segmentation and Retrievalcited by 18
- 2024Learning Inclusion Matching for Animation Paint Bucket Colorizationcited by 14
- 2024Low-Resource Vision Challenges for Foundation Modelscited by 13
- 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2025: 9; 2026: 5.
14 papers.
Sort:
By area:
- 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 14
- 2025OmniPSD: Layered PSD Generation with Diffusion Transformercited by 13
- 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generationcited by 9
- 2025Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceiverscited by 5
- 2025OmniRefiner: Reinforcement-Guided Local Diffusion Refinementcited by 5
- 2025NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editingcited by 4
- 2026GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflowscited by 4
- 2026SIGMA: Selective-Interleaved Generation with Multi-Attribute Tokenscited by 2
- 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignmentcited by 2
- 2025Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoisingcited by 1
- 2025Image Generation as a Visual Planner for Robotic Manipulationcited by 1
- 2025TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidancecited by 1
2 more
- 2026EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing8 May 2026
- 2026ID-Sim: An Identity-Focused Similarity Metric6 Apr 2026
- 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment19 Mar 2026
- 2026GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflows12 Mar 2026
- 2026SIGMA: Selective-Interleaved Generation with Multi-Attribute Tokens7 Feb 2026
- 2025Mitty: Diffusion-based Human-to-Robot Video Generation19 Dec 2025
- 2025OmniPSD: Layered PSD Generation with Diffusion Transformer10 Dec 2025
- 2025TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance1 Dec 2025
- 2025Image Generation as a Visual Planner for Robotic Manipulation29 Nov 2025
- 2025NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editing28 Nov 2025
- 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generation27 Nov 2025
- 2025OmniRefiner: Reinforcement-Guided Local Diffusion Refinement25 Nov 2025
- 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 14
- 2025OmniPSD: Layered PSD Generation with Diffusion Transformercited by 13
- 2025Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceiverscited by 5
- 2025OmniRefiner: Reinforcement-Guided Local Diffusion Refinementcited by 5
- 2025NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editingcited by 4
- 2026SIGMA: Selective-Interleaved Generation with Multi-Attribute Tokenscited by 2
- 2025Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoisingcited by 1
- 2026SIGMA: Selective-Interleaved Generation with Multi-Attribute Tokens7 Feb 2026
- 2025Mitty: Diffusion-based Human-to-Robot Video Generation19 Dec 2025
- 2025OmniPSD: Layered PSD Generation with Diffusion Transformer10 Dec 2025
- 2025NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editing28 Nov 2025
- 2025OmniRefiner: Reinforcement-Guided Local Diffusion Refinement25 Nov 2025
- 2025Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers24 Nov 2025
- 2025Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising5 Nov 2025
- 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignmentcited by 2
- 2025Image Generation as a Visual Planner for Robotic Manipulationcited by 1
- 2026ID-Sim: An Identity-Focused Similarity Metriccited by 1
- 2026EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editingcited by 1
- 2026EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing8 May 2026
- 2026ID-Sim: An Identity-Focused Similarity Metric6 Apr 2026
- 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment19 Mar 2026
- 2025Image Generation as a Visual Planner for Robotic Manipulation29 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2026Contextualized Visual Personalization in Vision-Language Models
- 2026Personalize Your Large Vision-language Models With In-context Prompt Tuning
- 2026FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
- 2024Teaching VLMs to Localize Specific Objects from In-context Examples
- 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Models
- 2026Personal Visual Context Learning in Large Multimodal Models
- 2023In-Context Learning Unlocked for Diffusion Models
- 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models