# Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Model

arXiv: https://arxiv.org/abs/2405.10316 · HTML: https://mlsanity.com/paper/2405.10316/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 16 May 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Visual Attribute Transfer through Deep Image Analogy](https://mlsanity.com/paper/1705.01088/) (2017)
- [DwNet: Dense warp-based network for pose-guided human video generation](https://mlsanity.com/paper/1910.09139/) (2019)
- [In-Context Learning Unlocked for Diffusion Models](https://mlsanity.com/paper/2305.01115/) (2023)
- [IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers](https://mlsanity.com/paper/2304.14400/) (2023)
- [ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation](https://mlsanity.com/paper/2308.00906/) (2023)
- [ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction](https://mlsanity.com/paper/2303.05063/) (2023)
- [Deep Generation of Face Images from Sketches](https://mlsanity.com/paper/2006.01047/) (2020)
- [CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing](https://mlsanity.com/paper/2307.08397/) (2023)
- [What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs](https://mlsanity.com/paper/2401.02411/) (2024)
- [CariMe: Unpaired Caricature Generation with Multiple Exaggerations](https://mlsanity.com/paper/2010.00246/) (2020)

## Built on by (most cited first, 30 of 30)

- [JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization](https://mlsanity.com/paper/2511.23002/) (2025)
- [ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation](https://mlsanity.com/paper/2502.09411/) (2025)
- [Personalized Vision via Visual In-Context Learning](https://mlsanity.com/paper/2509.25172/) (2025)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [Visual Prompting for One-shot Controllable Video Editing without Inversion](https://mlsanity.com/paper/2504.14335/) (2025)
- [PairEdit: Learning Semantic Variations for Exemplar-based Image Editing](https://mlsanity.com/paper/2506.07992/) (2025)
- [LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair](https://mlsanity.com/paper/2411.19156/) (2024)
- [Omni-3DEdit: Generalized Versatile 3D Editing in One-Pass](https://mlsanity.com/paper/2603.17841/) (2026)
- [Spanning the Visual Analogy Space with a Weight Basis of LoRAs](https://mlsanity.com/paper/2602.15727/) (2026)
- [INRetouch: Context Aware Implicit Neural Representation for Photography Retouching](https://mlsanity.com/paper/2412.03848/) (2024)
- [IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting](https://mlsanity.com/paper/2501.15641/) (2025)
- [Difference Inversion: Interpolate and Isolate the Difference with Token Consistency for Image Analogy Generation](https://mlsanity.com/paper/2506.07750/) (2025)
- [Is Visual in-Context Learning for Compositional Medical Tasks within Reach?](https://mlsanity.com/paper/2507.00868/) (2025)
- [Splatent: Splatting Diffusion Latents for Novel View Synthesis](https://mlsanity.com/paper/2512.09923/) (2025)
- [Universal Few-Shot Spatial Control for Diffusion Models](https://mlsanity.com/paper/2509.07530/) (2025)
- [ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Control](https://mlsanity.com/paper/2603.14209/) (2026)
- [FlowFixer: Towards Detail-Preserving Subject-Driven Generation](https://mlsanity.com/paper/2602.21402/) (2026)
- [Language-Free Generative Editing from One Visual Example](https://mlsanity.com/paper/2603.25441/) (2026)
- [Delta-Adapter: Scalable Exemplar-Based Image Editing with Single-Pair Supervision](https://mlsanity.com/paper/2605.07940/) (2026)
- [Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy](https://mlsanity.com/paper/2412.12463/) (2024)
- [One-shot Embroidery Customization via Contrastive LoRA Modulation](https://mlsanity.com/paper/2509.18948/) (2025)
- [Cycle-Consistent Tuning for Layered Image Decomposition](https://mlsanity.com/paper/2602.20989/) (2026)
- [Abstraction in Style: Beyond Texture and Color](https://mlsanity.com/paper/2603.29924/) (2026)
- [Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective](https://mlsanity.com/paper/2604.05651/) (2026)
- [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026)
- [MirrorPPR: Exemplar-Based Portrait Photo Retouching](https://mlsanity.com/paper/2606.29308/) (2026)
- [Hidden-Shot: Towards One-Shot Task Generalization for Low-Level Vision Generalist Models](https://mlsanity.com/paper/2607.01535/) (2026)
- [Tree-of-Thoughts Reasoning for Text-to-Image In-Context Learning](https://mlsanity.com/paper/2607.07117/) (2026)
- [TransPhy: Visual In-Context Learning for Physically Grounded Image Editing](https://mlsanity.com/paper/2608.24119/) (2026)
- [Unifying Video Tasks via Spatiotemporal Analogy](https://mlsanity.com/paper/2609.33935/) (2026)

## Abstract (arXiv)

Visual In-Context Learning (ICL) has emerged as a promising research area due to its capability to accomplish various tasks with limited example pairs through analogical reasoning. However, training-based visual ICL has limitations in its ability to generalize to unseen tasks and requires the collection of a diverse task dataset. On the other hand, existing methods in the inference-based visual ICL category solely rely on textual prompts, which fail to capture fine-grained contextual information from given examples and can be time-consuming when converting from images to text prompts. To address these challenges, we propose Analogist, a novel inference-based visual ICL approach that exploits both visual and textual prompting techniques using a text-to-image diffusion model pretrained for image inpainting. For visual prompting, we propose a self-attention cloning (SAC) method to guide the fine-grained structural-level analogy between image examples. For textual prompting, we leverage GPT-4V's visual reasoning capability to efficiently generate text prompts and introduce a cross-attention masking (CAM) operation to enhance the accuracy of semantic-level analogy guided by text prompts. Our method is out-of-the-box and does not require fine-tuning or optimization. It is also generic and flexible, enabling a wide range of visual tasks to be performed in an in-context manner. Extensive experiments demonstrate the superiority of our method over existing approaches, both qualitatively and quantitatively.