# X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models

arXiv: https://arxiv.org/abs/2412.01824 · HTML: https://mlsanity.com/paper/2412.01824/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 95 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://mlsanity.com/paper/2409.04429/) (2024)
- [PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction](https://mlsanity.com/paper/2410.17247/) (2024)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Making LLaMA SEE and Draw with SEED Tokenizer](https://mlsanity.com/paper/2310.01218/) (2023)
- [Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning](https://mlsanity.com/paper/2309.02591/) (2023)
- [Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation](https://mlsanity.com/paper/2409.04410/) (2024)
- [Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining](https://mlsanity.com/paper/2408.02657/) (2024)
- [HART: Efficient Visual Generation with Hybrid Autoregressive Transformer](https://mlsanity.com/paper/2410.10812/) (2024)
- [Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models](https://mlsanity.com/paper/2409.10695/) (2024)

## Built on by (most cited first, 15 of 15)

- [SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience](https://mlsanity.com/paper/2508.04700/) (2025)
- [RelightVid: Temporal-Consistent Diffusion Model for Video Relighting](https://mlsanity.com/paper/2501.16330/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025)
- [RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models](https://mlsanity.com/paper/2503.10406/) (2025)
- [ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing](https://mlsanity.com/paper/2506.19848/) (2025)
- [MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation](https://mlsanity.com/paper/2508.11433/) (2025)
- [CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning](https://mlsanity.com/paper/2508.20096/) (2025)
- [UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models](https://mlsanity.com/paper/2509.21760/) (2025)
- [DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning](https://mlsanity.com/paper/2601.21716/) (2026)
- [SpotEdit: Evaluating Visually-Guided Image Editing Methods](https://mlsanity.com/paper/2508.18159/) (2025)
- [T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs](https://mlsanity.com/paper/2511.16107/) (2025)
- [UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling](https://mlsanity.com/paper/2607.24157/) (2026)
- [Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning](https://mlsanity.com/paper/2608.23570/) (2026)

## Abstract (arXiv)

In-context generation is a key component of large language models' (LLMs) open-task generalization capability. By leveraging a few examples as context, LLMs can perform both in-domain and out-of-domain tasks. Recent advancements in auto-regressive vision-language models (VLMs) built upon LLMs have showcased impressive performance in text-to-image generation. However, the potential of in-context learning for general image generation tasks remains largely unexplored. To address this, we introduce X-Prompt, a purely auto-regressive large-vision language model designed to deliver competitive performance across a wide range of both seen and unseen image generation tasks, all within a unified in-context learning framework. X-Prompt incorporates a specialized design that efficiently compresses valuable features from in-context examples, supporting longer in-context token sequences and improving its ability to generalize to unseen tasks. A unified training task for both text and image prediction enables X-Prompt to handle general image generation with enhanced task awareness from in-context examples. Extensive experiments validate the model's performance across diverse seen image generation tasks and its capacity to generalize to previously unseen tasks.